SlowGuess/NAACL_2021_OCR huggingface dataset:
Combination. We first convert MultiNLI and Semantic Fragments into 2-way (entailment vs not_entsailment) NLI problems by collapsing both contradiction and neutral labels into not_entsailment, and resampling such that the dataset is balanced between the 2 label classes. To evaluate on QANLI, we use the RACE QA dataset and transform it into NLI as in Demszky et al. (2018). For RTE, we create a test set ourselves by randomly sampling examples from the RTE dataset provided by Wang et al. (2019). Dataset statistics can be found in Table 7.
GLUE-SciTail. We use MultiNLI, RTE, QNLI and SNLI as training data, following a similar procedure to convert 3-way NLI datasets into 2-way NLI. For evaluation, we use SciTail. Dataset statistics are in Table 8.
A.2.2 Training Details
Hyperparameters for all MAML models can be found in Table 9. We implement MAML in Pytorch using the higher library (Grefenstette et al., 2019). We take the BERT-base implementation from the huggingface library (Wolf et al., 2019) as
Task	Syntactic Templates
1	ce_adverb, cn_adverb
2	ce_embedding_under Verb, cn_embedding_under Verb
3	ce_conjunction, cn_disjunction
4	ce_after_since_clause, cn_after_if_clause
5	ce_embedding_under_since, cn_embedding_under_if
6	le_conjunction, ln_conjunction
7	le.passive, ln.passive
8	le(relative_clause, ln-relative_clause
9	le_around_prepositional Phrase, ln_preposition
10	le_around-relative_clause, lnsubject/object_swap
11	se_PP_on_obj, sn_PP_onsubject
12	se-relative_clause_on_obj, sn-relative_clause_onsubject
13	se_adjective, sn_NP/S
14	se_conjunction, sn_NP/Z
15	se_understood_object, sn_past_participle
Table 6: The 15 reasoning categories constructed from 30 HANS syntactic templates. For each reasoning category, we select 2 syntactic templates corresponding to entailment and not_entailment labels, giving us 15 binary classification tasks.
Dataset	#examples
Training Datasets
MultiNLI	261798
DNC	440456
Semantic Fragments	33170
Test Datasets
RTE	554
QANLI	1990
Table 7: Dataset statistics for COMBINEDNLI
the parameterization for  $h_{\theta}$ , which has 110 million parameters
DRECA. We first finetune BERT on the entire training dataset for 5 epochs. Then, we embed each example by concatenating the embedding at the [CLS] token along with the mean pooled representation of the premise and the hypothesis to get a 2304-dimensional vector. Next, we apply PCA to select a subset of dimensions that explain  $99\%$  of the variance. We then apply k-means clustering after standardizing the resulting embeddings.
Dataset	#examples
Training Datasets
MultiNLI	261798
SNLI	366832
QNLI	104732
RTE	2482
Test Dataset
SciTail	2126
Table 8: Dataset statistics for GLUE-SciTail
Hyperparameter	Value
inner loop learning rate	5e-5
outer loop learning rate	5e-5
inner loop adaptation steps	10
inner / outer loop optimizer	Adam
max number of iterations	20000
episode size	32
episode batch size	5
Table 9: Hyperparameters for MAML based models.
B Discovering Reasoning Categories in 2D Sine Wave Regression
To discover latent reasoning categories for the 2D Sine Wave Regression dataset, we train a feedforward neural net (paramaterized similarly as  $h_{\theta}$ ) on the union of all the datasets, and use the final layer representation to cluster examples. We then use these clusters instead of the true reasoning categories to augment the original task distribution.
We now show learning curves on held out test tasks in Fig. 6. As expected, MAML-BASE fails to adapt to new reasoning categories, indicating that it was unable to acquire the required skill from its training tasks. On the other hand, MAML-ORACLE is able to adapt very well, which confirms our hypothesis that a large number of high quality tasks helps. Finally, we see that using MAML trained on the augmented task distribution is able to match the performance of the oracle.
Figure 6: Learning curves on the 2D sine-wave regression task. We observe that the oracle meta-learner outperforms the baseline, and our proposed approach is able to bridge the gap.