SlowGuess/NAACL_2021_OCR huggingface dataset:
Shikhar Murty
{smurty,thashim,manning}@cs.stanford.edu
Meta-learning promises few-shot learners that quickly adapt to new distributions by repurposing knowledge acquired from previous training. However, we believe meta-learning has not yet succeeded in NLP due to the lack of a well-defined task distribution, leading to attempts that treat datasets as tasks. Such an ad hoc task distribution causes problems of quantity and quality. Since there's only a handful of datasets for any NLP problem, meta-learners tend to overfit their adaptation mechanism and, since NLP datasets are highly heterogeneous, many learning episodes have poor transfer between their support and query sets, which discourages the meta-learner from adapting. To alleviate these issues, we propose DRECA (Decomposing datasets into Reasoning Categories), a simple method for discovering and using latent reasoning categories in a dataset, to form additional high quality tasks. DRECA works by splitting examples into label groups, embedding them with a finetuned BERT model and then clustering each group into reasoning categories. Across four few-shot NLI problems, we demonstrate that using DRECA improves the accuracy of meta-learners by  $1.5 - 4\%$ .
A key desideratum for human-like understanding is few-shot adaptation. Adaptation is central to many NLP applications since new concepts and words appear often, leading to distribution shifts. People can effortlessly deal with these distribution shifts by learning these new concepts quickly and we would like our models to have similar capabilities. While finetuning large pre-trained transformers is one way to facilitate this adaptation, this procedure requires thousands of samples where humans might require only a few.
Can these pre-trained transformers be made to achieve few-shot adaptation? One promising direction is meta-learning (Schmidhuber, 1987; Ben
Figure 1: Overview of our approach. We embed all examples with BERT, and then cluster within each label group separately (red and green correspond to entailment and not_ entailment respectively). Then, we group clusters from distinct label groups to form tasks.
gio et al., 1997). Meta-learning promises few-shot classifiers that can adapt to new tasks by repurposing skills acquired from training tasks. An important prerequisite for successful application of meta-learning is a task-distribution from which a large number of tasks can be sampled to train the meta-learner. While meta-learning is very appealing, applications in NLP have thus far proven challenging due to the absence of a well-defined set of tasks that correspond to re-usable skills. This has led to less effective ad hoc alternatives, like treating entire datasets as tasks.
Treating entire datasets as tasks has two major issues. The first issue is learner overfitting (Rajendran et al., 2020), where a meta-learner overfits its adaptation mechanism to the small number of training tasks, since there's only a small number of supervised datasets available for any NLP problem. Second, the heterogeneity of NLP datasets can lead to learning episodes that encourage memorization overfitting (Yin et al., 2020; Rajendran et al., 2020), a phenomenon where a meta-learner ignores the support set, and doesn't learn to adapt.
To improve the quality and quantity of tasks, we propose the simple approach of Decomposing
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1113-1125 June 6-11, 2021. ©2021 Association for Computational Linguistics
datasets into Reasoning Categories or DRECA. DRECA is a meta data augmentation strategy that takes as input a set of tasks (entire datasets), and then decomposes them to approximately recover some of the latent reasoning categories underlying these datasets, such as various syntactic constructs within a dataset, or semantic categories such as quantifiers and negation. These reasoning categories are then used to construct additional few-shot classification tasks, augmenting the original task distribution. We illustrate these steps in Fig. 1. DRECA first embeds the examples using a BERT model finetuned over all the datasets. We then run k-means clustering over these representations to produce a refinement of the original tasks.
Experiments demonstrate the effectiveness of our simple approach. As a proof of concept, we adapt the classic sine-wave regression problem from Finn et al. (2017) to mimic the challenges of the NLP setting, and observe that standard meta-learning procedures fail to adapt. However, a model that meta-learns over the underlying reasoning types shows a substantial improvement. Then, we consider the problem of natural language inference (NLI). We show that meta-learners augmented with DRECA improve over baselines by 1.5–4 accuracy points across four separate NLI few-shot problems without requiring domain-specific engineering or additional unlabeled data.
Few-shot learning in NLP. The goal of learning from few examples has been studied for various NLP applications. Common settings include few-shot adaptation to new relations (Han et al., 2018), words (Holla et al., 2020), domains (Bao et al., 2020; Yu et al., 2018; Geng et al., 2019), and language pairs (Gu et al., 2018). Since these applications come with well-defined task distributions, they do not have the same overfitting challenges. On the other hand, many works deal with few-shot adaptation in settings with no clear task distribution (Dou et al., 2019; Bansal et al., 2020a) but do not address meta-overfitting, and thus are complementary to our work.
Overfitting and Task Augmentation. The memorization problem in meta-learning is studied in Yin et al. (2020) who propose a meta-regularizer to mitigate memorization overfitting, but don't study learner overfitting. Task augmentation for mitigating overfitting in meta-learners is
first studied in Rajendran et al. (2020) in the context of few-shot label adaptation. Hsu et al. (2019) propose CACTUs, a clustering-based approach for unsupervised meta-learning in the context of few-shot label adaptation for images. While also based on clustering, CACTUs creates meta-learning tasks where the goal is to predict cluster membership of images, whereas our work is focused on using clusters to subdivide pre-existing tasks for mitigating meta-overfitting in NLP. Most closely related to our work is the SMLMT method from Bansal et al. (2020b). SMLMT creates new self-supervised tasks that improve meta-overfitting but this does not directly address the dataset-as-tasks problem we identify. In contrast, we focus on using clustering as a way to subdivide and fix tasks that already exist. This approach allows us to mitigate meta-overfitting without additional unlabeled data. In Section 6, we compare our model against SMLMT, and demonstrate comparable or better performance.
3.1 NLI
We consider the problem of Natural Language Inference or NLI (MacCartney and Manning, 2008; Bowman et al., 2015), also known as Recognising Textual Entailment (RTE) (Dagan et al., 2005). Given a sentence pair  $x = (p, h)$  where  $p$  is referred to as the premise sentence, and  $h$  is the hypothesis sentence, the goal is to output a binary label  $\hat{y} \in \{0, 1\}$  indicating whether the hypothesis  $h$  is entailed by the premise  $p$  or not. For instance, the sentence pair (The dog barked, The animal barked) is classified as entailed, whereas the sentence pair (The dog barked, The labrador barked) would be classified as not entailed. As shown in Table 1, NLI datasets typically encompass a broad range of linguistic phenomena. Apart from the reasoning types shown in Table 1, examples may also vary in terms of their genre, syntax, annotator writing style etc. leading to extensive linguistic variability. Taken together, these factors of variation make NLI datasets highly heterogeneous.
3.2 Meta-Learning
The goal of meta-learning is to output a metalearner  $f\colon (S_i,x_q^i)\mapsto \hat{y}$  that takes as input a support set  $S_{i}$  of labeled examples and a query point
<sup>1</sup>Since many of the NLI datasets we experiment with are 2-way NLI, we choose this formulation instead of 3-way NLI.
Example	Reasoning Category
A boy with the green jacket went back ⇒ A boy went back	Restrictive Modifiers
A white rabbit ran ⇒ A rabbit ran	Intersective Adjectives
Bill is taller than Jack ⇒ Jack is taller than Bill	Comparatives
The dog barked ⇒ The dog did not bark	Negation
The man went to the restaurant since he was hungry ⇒ The man was hungry	Coreference Resolution
Bill is taller than Jack ⇒ Jack is not taller than Bill	Negated Comparatives
Table 1: Some common reasoning types within NLI. These can also be composed to create new types.
$x_{q}^{i}$  and returns a prediction  $\hat{y}$ . In the usual meta-learning setting, these support and query sets are defined as samples from a task  $\mathcal{T}^i$ , which is a collection of labeled examples  $\{(x^i,y^i)\}$ . In  $N$ -way  $k$ -shot adaptation, each  $\mathcal{T}^i$  is an  $N$ -way classification problem, and  $f$  is given  $k$  examples per label to adapt. A simple baseline for meta-learning is to train a supervised model on labeled data from training tasks, and then finetune it at test time on the support set. This can be powerful, but is ineffective for very small support sets. A better alternative is episodic meta-learning, which explicitly trains models to adapt using training tasks
Episodic Training. In the standard setup for training episodic meta-learners, we are given a collection of training tasks. We assume that both train and test tasks are i.i.d. draws from a task distribution  $p(T)$ . For each training task  $\mathcal{T}_i^{\mathrm{tr}} \sim p(\mathcal{T})$ , we create learning episodes which are used to train the meta-learner. Each learning episode consists of a support set  $S_{i} = \{(x_{s}^{i},y_{s}^{i})\}$  and a query set  $Q_{i} = \{(x_{q}^{i},y_{q}^{i})\}$ . The goal of episodic meta-learning is to ensure that the meta-learning loss  $\mathcal{L}(f(S_i,x_q^i),y_q^i)$  is small on training tasks  $T_{i}^{\mathrm{tr}}$ . Since train tasks are i.i.d. with test tasks, this results in meta-learners that achieve low loss at test time.
Several algorithms have been proposed for meta-learning that follow this general setup, such as Matching Networks (Vinyals et al., 2016), MANN (Santoro et al., 2016), Prototypical Networks (Snell et al., 2017) and MAML (Finn et al., 2017). In this work, we use MAML as our meta-learner.
MAML. In MAML, the meta-learner  $f$  takes the form of gradient descent on a model  $h_{\theta} \colon x \mapsto y$  using the support set,
$$
f \left(S _ {i}, x _ {i} ^ {q}\right) = h _ {\theta_ {i} ^ {\prime}} \left(x _ {i} ^ {q}\right) \tag {1}
$$
where  $\theta_i^\prime$  denotes task-specific parameters obtained after gradient descent. The goal of MAML is to
produce an initialization  $\theta$ , such that after performing gradient descent on  $h_{\theta}$  using  $\mathcal{S}_i$ , the updated model  $h_{\theta_i'}$  can make accurate predictions on  $\mathcal{Q}_i$ . MAML consists of an inner loop and an outer loop. In the inner loop, the support set  $\mathcal{S}_i$  is used to update model parameters  $\theta$ , to obtain task-specific parameters  $\theta_i'$ ,
$$
\theta_ {i} ^ {\prime} = \theta - \alpha \nabla_ {\theta} \sum_ {\left(x _ {s} ^ {i}, y _ {s} ^ {i}\right) \in \mathcal {S} _ {i}} \mathcal {L} \left(h _ {\theta} \left(x _ {s} ^ {i}\right), y _ {s} ^ {i}\right). \tag {2}
$$
These task-specific parameters are then used to make predictions on  $\mathcal{Q}_i$ . The outer loop takes gradient steps over  $\theta$  such that task-specific parameters  $\theta_i'$  perform well on  $\mathcal{Q}_i$ . Since  $\theta_i'$  is itself a differentiable function of  $\theta$ , we can perform this outer optimization using gradient descent,
$$
\theta \leftarrow \operatorname {O p t} \left(\theta , \nabla_ {\theta} \sum_ {\left(x _ {q} ^ {i}, y _ {q} ^ {i}\right) \in \mathcal {Q} _ {i}} \mathcal {L} \left(h _ {\theta_ {i} ^ {\prime}} \left(x _ {q} ^ {i}\right), y _ {q} ^ {i}\right)\right). \tag {3}
$$
where Opt is an optimization algorithm typically chosen to be Adam. The outer loop gradient is typically computed in a mini-batch fashion by sampling a batch of episodes from distinct training tasks. The gradient  $\nabla_{\theta}\mathcal{L}(h_{\theta^{\prime}_i}(x_q^i),y_q^i)$  involves back-propagation through the adaptation step which requires computing higher order gradients. This can be computationally expensive so a first order approximation (FoMAML),
$$
\nabla_ {\theta} \mathcal {L} (h _ {\theta_ {i} ^ {\prime}} (x _ {q} ^ {i}), y _ {q} ^ {i}) \approx \nabla_ {\theta_ {i} ^ {\prime}} \mathcal {L} (h _ {\theta_ {i} ^ {\prime}} (x _ {q} ^ {i}), y _ {q} ^ {i}) \qquad (4)
$$
is often used instead (Finn et al., 2017).
3.3 Meta-Learning for NLI
As mentioned earlier, training tasks in NLP are often entire datasets, leading to a small number of heterogeneous training tasks. Thus, to train a meta-learner for NLI, our training tasks  $\mathcal{T}_i^{\mathrm{tr}}$  are NLI datasets. At test time, we are given new datasets that we must adapt to, given a support set of randomly drawn examples from the dataset.
Meta Overfitting. Consider learning episodes sampled from an NLI dataset (Table 2). NLI datasets consist of a wide range of linguistic phenomena, and so we expect an episode to be comprised of a diverse set of reasoning categories. Such heterogeneous episodes can lead to scenarios where the support and query sets do not have any overlap in reasoning skills, causing the model to ignore the support set. This is known as memorization overfitting. Moreover, since we have a limited number of datasets, the meta-learner is exposed to a very small number of tasks at meta-training time causing it to generalize poorly to test tasks. This is known as learner overfitting (Rajendran et al., 2020).
NLI Example	Reasoning Category
Everyone has visited every person ⇒ Jeff didn’t visit Byron	Negation, Quantifier
Generally, LC mail is lighter than AO mail ⇒ AO mail is almost always heavier than LC mail	Comparative, Quantifier
They’ve had their house that long ⇒ They don’t own the house and have never lived there	Negation
Then he strolled gently in the opposite direction ⇒ He wasn’t walking in the same direction	Negation
Query
A white rabbit ran ⇒ A rabbit ran	Intersective adjective
Table 2: Illustration of an episode sampled from a heterogeneous task. We can observe that there is no overlap between the support and query reasoning categories, leading to limited transfer.
4 An Illustration of Overfitting in Meta-Learning
We illustrate meta overfitting challenges by modifying the classic sine-wave toy example for meta-learning from Finn et al. (2017).
Dataset. Consider the sine-wave regression problem from Finn et al. (2017) where each task corresponds to learning a sine wave mapping with a fixed amplitude and phase offset. As shown in Fig. 2(a), each support and query set consists of points drawn from the same sine wave mapping. The key observation here is that since support and query examples are drawn from the same mapping, we might expect a meta-learner to use the support set for adaptation. In the NLP case, since tasks are heterogeneous, support and query examples may belong to different reasoning categories. We instantiate this by letting support and query points come from different sine waves (Fig. 2(b)).
More formally, our construction consists of multiple datasets. Each dataset is defined as a unit
(a) 1D sine wave regression (Finn et al., 2017). Each task is a sine-wave with a fixed amplitude and phase offset.
(b) Three datasets from our 2D sine wave regression. Each dataset is a unit square with multiple reasoning categories; A reasoning category is a distinct sinusoid along a ray that maps  $x = (x_{1}, x_{2})$  to the value of the sine-wave  $y$  at that point.
Figure 2: Comparing the classic 1D sine wave regression with our setting. For a randomly sampled episode, red dots mark support examples and the green square marks a query example. Notice how in 2(a), the support and query come from the same sine wave while in 2(b) they often come from different sine waves. This makes adaptation challenging, leading to memorization overfitting.
square sampled from a  $10 \times 10$  grid over  $x_{1} = [-5,5]$  and  $x_{2} = [-5,5]$ . Within each dataset, we construct multiple reasoning categories by defining each reasoning category to be a sine wave with a distinct phase offset. This is illustrated in Fig. 2(b) where each unit square represents a dataset, and sine waves along distinct rays correspond to reasoning categories. The target label  $y$  for the regression task is defined for each category by a randomly sampled phase  $\phi \in [0.1,2\pi]$  and  $y = \sin (\| x - \lfloor x\rfloor \| _2 - \phi)$ . At meta-training time, we sample a subset of these 100 squares as our training datasets, and then evaluate few-shot adaptation to reasoning categories from held out datasets at meta-test time.
Experiments. We use similar hyperparameters as Finn et al. (2017) elaborated in Appendix A.1.
We start by considering MAML-BASE, a meta-learner that is trained directly over a dataset-based task distribution. Concretely, we define each training task as a dataset and randomly sample episodes to train the meta-learner. Note that since episodes are drawn uniformly at random from an entire dataset, we expect support and query sets to often contain points from disjoint reasoning categories
(Fig. 2(b)), making adaptation infeasible. Thus, we expect pre and post adaptation losses to be similar, which is indeed reflected in the learning curves in Fig. 3(a). We observe that the orange and blue lines, corresponding to pre and post adaptation losses respectively, almost overlap. In other words, the meta-learner ignores the support set entirely. This is what we mean by memorization overfitting.
Next we consider MAML-ORACLE, a metalearner that is trained on tasks based on the underlying reasoning categories—distinct sine waves. Consequently, support and query sets are both drawn from the same sine wave, similar to Finn et al. (2017) making adaptation feasible. From Fig. 3(b), we observe large gaps between pre and post adaptation losses which indicates that memorization overfitting has been mitigated.
(b)
Figure 3: Learning curves for MAML-BASE (a) and MAML-ORACLE (b). The lack of a gap between pre-adaptation (orange) and post-adaptation (blue) losses for MAML-BASE indicates strong memorization overfitting. A big gap for MAML-ORACLE indicates that this model learns to adapt.
These experiments confirm our hypothesis about the challenges of meta-learning with heterogeneous task distributions. Since NLI datasets require a wide range of skills, we might expect similar challenges on few-shot NLI as well.
5 DRECA
In this section, we introduce our approach for extracting reasoning categories for NLI. The key observation here is that high quality sentence pair representations, such as those obtained from a finetuned BERT model, can bring out the microstructure of NLI datasets. Indeed, the fact that pretrained transformers can be used to create meaningful clusters has been shown in other recent works (c.f. Aharoni and Goldberg (2020); Joshi et al. (2020)).
At a high level, the goal of DRECA is to take a heterogeneous task (such as a dataset) and produce a decomposed set of tasks. In doing so, we hope to
obtain a large number of relatively homogeneous tasks that can prevent meta overfitting.
Given a training task  $\mathcal{T}_i^{\mathrm{tr}}$ , we first group examples by their labels, and then embed examples within each group with an embedding function EMBED(). Concretely, for each  $N$ -way classification task  $\mathcal{T}_i^{\mathrm{tr}}$  we form groups  $g_l^i = \{(EMBED(x_i^p), y_i^p) \mid y_i^p = l\}$ . Then, we proceed to refine each label group into  $K$  clusters via k-means clustering to break down  $\mathcal{T}_i^{\mathrm{tr}}$  into groups  $\{C^j(g_l^i)\}_{j=1}^K$  for  $l = 1, 2, \ldots, N$ .
These cluster groups can be used to produce  $K^N$  potential DRECA tasks. Each task is obtained by choosing one of  $K$  clusters for each of the  $N$  label groups, and taking their union. At meta-training time, learning episodes are sampled uniformly at random from DRECA tasks with a probability  $\lambda$  and from one of the original tasks with probability  $1 - \lambda$ . Since our clustering procedure is based on finetuned BERT vectors, we expect the resulting clusters to roughly correspond to distinct reasoning categories. Indeed, when the true reasoning categories are known, we show in Section 7.2 that DRECA yields clusters that recover these reasoning categories almost exactly.
6 NLI Experiments
We evaluate DRECA on 4 NLI few-shot learning problems which we describe below (more details in Appendix A.2.1). The first problem is based on synthetic data, while the other 3 problems are on real datasets and hence a good demonstration of the utility of our proposal.
HANS-FEWSHOT is a few-shot classification problem over HANS (McCoy et al., 2019), a synthetic diagnostic dataset for NLI. Each example in HANS comes from a hand-designed syntactic template which is associated with a fixed label (entailment or not_entailment). The entire dataset consists of 30 such templates which we use to define 15 reasoning categories. We then hold out 5 of these for evaluation, and train on the remaining 10. While this is a simple setting, it allows us to compare DRECA against an "oracle" with access to the underlying reasoning categories.
2Note that we do not instantiate the  $K^N$  tasks. Instead, we simply sample an episode from random chosen clusters from each label group.
COMBINEDNLI consists of a combination of 3 NLI datasets—MultiNLI (Williams et al., 2018), Diverse Natural Language Inference Collection (DNC; Poliak et al. (2018)) and Semantic Fragments (Richardson et al., 2020) for training. These training datasets cover a broad range of NLI phenomena. MultiNLI consists of crowdsourced examples, DNC consists of various semantic annotations from NLP datasets re-cast into NLI and Semantic fragments is a synthetic NLI dataset covering logical and monotonicity reasoning. Our objective is to train a single meta-learner that can then be used to make predictions on diverse NLP problems recast as NLI. To this end, we evaluate models trained on COMBINEDNLI on 2 datasets. In COMBINEDNLI-RTE, we evaluate on the RTE datasets (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) as provided in GLUE (Wang et al., 2019). The RTE datasets consist of various IE and QA datasets recast as NLI. Second, we consider the QANLI dataset (Demszky et al., 2018) which recasts question answering into NLI. In particular, we consider RACE (Lai et al., 2017) and use gold annotations provided in Demszky et al. (2018) to transform it into an NLI dataset.
GLUE-SciTail where we train on all NLI datasets from GLUE (Wang et al., 2019) and evaluate on SciTail (Khot et al., 2018). This setting is comparable to Bansal et al. (2020b) with the difference that we only meta-train on the NLI subset of GLUE, whereas they meta-train on all GLUE tasks. We follow the same evaluation protocol as Bansal et al. (2020b) and report 2-way 4-shot accuracy.
6.2 Models
Non-Episodic Baselines. All non-episodic baselines train  $h_{\theta}$  on the union of all examples from each  $T_{i}^{\mathrm{tr}}$ . In MULTITASK (FINETUNE), we additionally finetune the trained model on the support set of each test task. In MULTITASK (K-NN), each query example in the test task is labeled according to the nearest neighbor of the example in the support set. Finally, in MULTITASK (FINETUNE + K-NN), we first finetune the trained model on the support set and then label each query example based on its nearest neighbor in the support set.
Episodic Meta-learners. MAML-BASE is a MAML model where every task corresponds to a dataset. In the HANS-FEWSHOT setting where underlying reasoning categories are known, we also