SlowGuess/NAACL_2022_OCR huggingface dataset:
Sewon Min $^{1,2}$  Mike Lewis $^{2}$  Luke Zettlemoyer $^{1,2}$  Hannaneh Hajishirzi $^{1,3}$
<sup>1</sup>University of Washington <sup>2</sup>Meta AI <sup>3</sup>Allen Institute for AI
{sewon,lsz,hannaneh}@cs.washington.edu mikelewis@fb.com
We introduce MetaICL (Meta-training for In-Context Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set of training tasks. This metatraining enables the model to more effectively learn a new task in context at test time, by simply conditioning on a few training examples with no parameter updates or task-specific templates. We experiment on a large, diverse collection of tasks consisting of 142 NLP datasets including classification, question answering, natural language inference, paraphrase detection and more, across seven different meta-training/target splits. MetaICL outperforms a range of baselines including in-context learning without meta-training and multi-task learning followed by zero-shot transfer. We find that the gains are particularly significant for target tasks that have domain shifts from the meta-training tasks, and that using a diverse set of the meta-training tasks is key to improvements. We also show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target task, and outperforms much bigger models with nearly 8x parameters. Finally, we show that MetaICL is complementary to human-written instructions, and the best performance can be achieved by combining both approaches.
Large language models (LMs) have recently been shown to be able to do in-context learning (Brown et al., 2020), where they learn a new task simply by conditioning on a few training examples and predicting which tokens best complete a test input. This type of learning is attractive because the model learns a new task through inference alone, without any parameter updates. However, performance significantly lags behind supervised finetuning, results are often high variance (Zhao et al., 2021; Perez
et al., 2021), and it can be difficult to engineer the templates that convert existing tasks to this format.
In this paper, we address these challenges by introducing MetaICL: Meta-training for In-Context Learning. MetaICL tunes a pretrained language model on a large set of tasks to learn how to in-context learn, and is evaluated on strictly new unseen tasks. Each meta-training example matches the test setup—it includes  $k + 1$  training examples from one task that will be presented together as a single sequence to the language model, and the output of the final example is used to calculate the cross-entropy training loss. Simply finetuning the model in this data setup directly leads to better in-context learning—the model learns to recover the semantics of the task from the given examples, as must be done for in-context learning of a new task at test time. This approach is related to recent work that uses multi-task learning for better zero-shot performance at test time (Khashabi et al., 2020; Zhong et al., 2021; Mishra et al., 2022; Wei et al., 2022; Sanh et al., 2022). However, MetaICL is distinct as it allows learning new tasks from  $k$  examples alone, without relying on a task reformating (e.g., reducing everything to question answering) or task-specific templates (e.g., converting different tasks to a language modeling problem).
We experiment on a large, diverse collection of tasks taken from Ye et al. (2021) and Khashabi et al. (2020), including 142 text classification, question answering, natural language inference and paraphrase detection datasets. We report seven different settings, all with no overlap between meta-training and target tasks. This leads to 52 unique target tasks in total, which is the largest among all recent related work to the best of our knowledge.
Experimental results show that MetaICL consistently outperforms baselines including (1) a variety of LM in-context learning baselines without metatraining (Brown et al., 2020; Zhao et al., 2021; Holtzman et al., 2021; Min et al., 2022), and (2)
Human Language Technologies, pages 2791 - 2809
multi-task learning followed by zero-shot transfer (Zhong et al., 2021; Wei et al., 2022; Sanh et al., 2022). Gains over multi-task zero-shot transfer are particularly significant when meta-training tasks and target tasks are dissimilar, e.g. there are large differences in task formats, domains, or required skills. This demonstrates that MetaICL enables the model to recover the semantics of the task in context during inference even when the target does not share similarities with meta-training tasks. MetaICL often gets close to (and sometimes beats) the performance of models trained with supervised finetuning on the target datasets, and perform as well as models with  $8\mathrm{x}$  parameters. We also perform extensive ablations to identify key ingredients for success of MetaICL such as the number and diversity of meta-training tasks. Finally, we demonstrate MetaICL without any templates is better than recent work using human-written natural instructions, while the best performance is achieved by combining both approaches. Code and data are publicly released at github.com/facebookresearch/MetaICL.
In-context learning Brown et al. (2020) propose to use a language model (LM) conditioned on a concatenation of training examples for few-shot learning with no parameter updates. It has been further improved by later work (Zhao et al., 2021; Holtzman et al., 2021; Min et al., 2022), showing promising results on a variety of tasks. However, in-context learning with an LM achieves poor performance when the target task is very different from language modeling in nature or the LM is not large enough. Moreover, it can have high variance and poor worst-case accuracy (Perez et al., 2021; Lu et al., 2021).
Our paper is based on the core idea of in-context learning by conditioning on training examples. We show that, by explicitly training on an in-context learning objective, MetaICL achieves substantial improvements even with smaller LMs.
Meta-training via multi-task learning Our work is broadly inspired by a large body of work in meta-learning (Vilalta and Drissi, 2002; Finn et al., 2017) and multi-task learning (Evgeniou and Pontil, 2004; Ruder, 2017). Prior work has shown that multi-task learning on a large collection of tasks leads to better performance on a new
task, either when tested zero-shot (Khashabi et al., 2020; Zhong et al., 2021; Mishra et al., 2022; Wei et al., 2022) or when further finetuned (Aghajanyan et al., 2021; Ye et al., 2021). In particular, the former is closely related to our work, as it eliminates the need for parameter updates on a target task. However, these zero-shot models are either limited to tasks sharing the same format as training tasks (e.g., a question answering format) (Khashabi et al., 2020; Zhong et al., 2021), or rely heavily on task-specific templates (Mishra et al., 2022; Wei et al., 2022; Sanh et al., 2022) which are difficult to engineer due to high variance in performance from very small changes (Mishra et al., 2021).
In this paper, we propose a meta-training method for better in-context learning that improves few-shot performance. We show that it effectively learns semantics of a new task with no manual effort, significantly outperforming zero-shot transfer methods. Furthermore, while Wei et al. (2022) show that meta-training helps only when the model has 68B or more parameters, our experiments demonstrate improvements with a much smaller model (770M).
Chen et al. (2022), concurrently to our work, propose meta-training for in-context learning. Our approach differs in a number of ways: we remove requirements of human-written templates or instructions, and include more diverse tasks, stronger baselines, and extensive experiments in much larger scale with many meta-training/target splits.
3 MetaICL
We introduce MetaICL: Meta-training for In-Context Learning. Table 1 provides an overview of the approach. The key idea is to use a multi-task learning scheme over a large collection of metatraining tasks, in order for the model to learn how to condition on a small set of training examples, recover the semantics of a task, and predict the output based on it. Following previous literature (Brown et al., 2020), the training examples are concatenated and provided as an single input to the model, which is feasible for  $k$ -shot learning (e.g.,  $k = 16$ ). At test time, the model is evaluated on an unseen target task that comes with  $k$  training examples, and inference directly follows the same data format as in meta-training.
<sup>1</sup>We show that MetaICL without instructions is still better than zero-shot transfer with instructions, but by using instructions, performance of MetaICL further improves (Section 5.2).
Meta-training	Inference
Task	C meta-training tasks	An unseen target task
Data given	Training examples Ti = {(xi, yi)}N1j=1, ∀i ∈ [1, C] (Ni ≫ k)	Training examples (x1, y1), ..., (xk, yk), Test input x
Objective	For each iteration,
1. Sample task i ∈ [1, C]
2. Sample k + 1 examples from Ti: (x1, y1), ..., (xk+1, yk+1)
3. Maximize P(yk+1|x1, y1, ..., xk, yk, xk+1)	argmaxc∈C P(c|x1, y1, ..., xk, yk, x)
3.1 Meta-training
The model is meta-trained on a collection of tasks which we call meta-training tasks. For every iteration, one meta-training task is sampled, and  $k + 1$  training examples  $(x_{1},y_{1}),\dots ,(x_{k + 1},y_{k + 1})$  are sampled from the training examples of the chosen task. We then supervise the model by feeding the concatenation of  $x_{1},y_{1},\dots ,x_{k},y_{k},x_{k + 1}$  to the model as an input and train the model to generate  $y_{k + 1}$  using a negative log likelihood objective. This simulates in-context learning at inference where the first  $k$  examples serve as training examples and the last  $(k + 1)$ -th example is regarded as the test example.
For a new target task, the model is given  $k$  training examples  $(x_{1},y_{1}),\dots ,(x_{k},y_{k})$  as well as a test input  $x$ . It is also given a set of candidates  $\mathcal{C}$  which is either a set of labels (in classification) or answer options (in question answering). As in meta-training, the model takes a concatenation of  $x_{1},y_{1},\dots ,x_{k},y_{k},x$  as the input, and compute the conditional probability of each label  $c_{i}\in \mathcal{C}$ . The label with the maximum conditional probability is returned as a prediction.
3.3 Channel MetaICL
We introduce a noisy channel variant of MetaICL called Channel MetaICL, following Min et al. (2022). In the noisy channel model,  $P(y|x)$  is reparameterized to  $\frac{P(x|y)P(y)}{P(x)} \propto P(x|y)P(y)$ . We follow Min et al. (2022) in using  $P(y) = \frac{1}{|\mathcal{C}|}$  and modeling  $P(x|y)$  which allows us to use the channel approach by simply flipping  $x_{i}$  and  $y_{i}$ . Specifically, at meta-training time, the model is given a concatenation of  $y_{1}, x_{1}, \dots, y_{k}, x_{k}, y_{k+1}$  and is
Table 1: Overview of MetaICL (Section 3). MetaICL uses the same in-context learning setup at both meta-training and inference. At meta-training time,  $k + 1$  examples for a task is sampled, where the last example acts as the test example and the rest  $k$  examples act as the training examples. Inference is the same as typical in-context learning where  $k$  labeled examples are used to make a prediction for a test input.
Meta-train	Target
Setting	# tasks	# examples	Setting	# tasks
HR	61	819,200	LR	26
Classification	43	384,022	Classification	20
Non-Classification	37	368,768
QA	37	486,143	QA	22
Non-QA	33	521,342
Non-NLI	55	463,579	NLI	8
Non-Paraphrase	59	496,106	Paraphrase	4
Table 2: Statistics of seven different settings. Each row indicates meta-training/target tasks for each setting. ‘# tasks’ in meta-training is equivalent to  $C$  in Table 1. For all settings, there is no overlap in tasks between meta-training and target. ‘HR’ and ‘LR’ indicate high resource and low resource, respectively. Datasets and the task ontology are taken from CROSSFIT (Ye et al., 2021) and UNIFIEDQA (Khashabi et al., 2020). Full datasets for each split are provided in Appendix A.
trained to generate  $x_{k + 1}$  .At inference,the model computes  $\operatorname {argmax}_{c\in \mathcal{C}}P(x|y_1,x_1,\dots ,y_k,x_k,c)$
We use a large collection of tasks taken from CROSSFIT (Ye et al., 2021) and UNIFIEDQA (Khashabi et al., 2020). We have 142 unique tasks in total, covering a variety of problems including text classification, question answering (QA), natural language inference (NLI) and paraphrase detection. All tasks are in English.
We experiment with seven distinct settings as shown in Table 2, where there is no overlap between the meta-training and target tasks. The number of unique target tasks in total is 52, which is significantly larger than other relevant work (Khashabi et al., 2020; Zhong et al., 2021; Mishra et al., 2022;
Method	Meta train	Target
train	# samples
LMs
0-shot	X	X	0
PMI 0-shot	X	X	0
Channel 0-shot	X	X	0
In-context	X	X	k
PMI In-context	X	X	k
Channel In-context	X	X	k
Meta-trained
Multi-task 0-shot	✓	X	0
Channel Multi-task 0-shot	✓	X	0
MetaICL (Ours)	✓	X	k
Channel MetaICL (Ours)	✓	X	k
Fine-tune
Fine-tune	X	✓	k
Fine-tune w/ meta-train	✓	✓	k
Wei et al., 2022; Sanh et al., 2022). Each target task is either classification or multi-choice, where a set of candidate options ( $\mathcal{C}$  in Table 1) is given.
$\mathbf{HR} \rightarrow \mathbf{LR}$  (High resource to low resource): We experiment with a setting where datasets with 10,000 or more training examples are used as meta-training tasks and the rest are used as target tasks. We think using high resource datasets for meta-training and low resource datasets as targets is a realistic and practical setting for few-shot learning.
$\mathbf{X} \rightarrow \mathbf{X}$  ( $\mathbf{X} = \{\text{Classification}, \mathbf{QA}\}$ ): We experiment with two settings with meta-training and target tasks sharing the task format, although with no overlap in tasks.
Non- $\mathbf{X} \rightarrow \mathbf{X}$  ( $\mathbf{X} = \{\text{Classification}, \mathbf{QA}, \mathbf{NLI}, \text{Paraphase}\}$ ): Lastly, we experiment with four settings where meta-training tasks do not overlap with target tasks in task format and required capabilities. These settings require the most challenging generalization capacities.
Each setting has a subset of target tasks with no domain overlap with any meta-training tasks (e.g., finance, poem, climate or medical). We report both on all target tasks or on target tasks with no domain overlap only. Full details of the settings and datasets with citations are provided in Appendix A.
Table 3: Summary of the baselines and MetaICL. 'train' indicates whether the model is trained with parameter updates, and '# samples' indicates the number of training examples used on a target task. Our baselines include a range of recently introduced methods (Holtzman et al., 2021; Zhao et al., 2021; Min et al., 2022; Wei et al., 2022) as described in Section 4.2.
[P]: Time Warner is the world's largest media and Internet company.
[H]: Time Warner is the world's largest company.
Labels: entailment, not_ entailment
Holtzman et al. (2021)
Input	[P] question: [H] true or false? answer:
Output	{true, false}
Wei et al. (2022)
Input	[P] Based on the paragraph above, can we conclude that [H]?
Output	{yes, no}
Ours
Input	[P] [H]
Output	{entailment, not_ entailment}
Table 4: Example input-output pairs for an NLI task. We show human-authored templates taken from prior work as references.
We compare MetaICL and Channel MetaICL with a range of baselines, as summarized in Table 3.
0-shot: We use a pretrained LM as it is and run zero-shot inference, following Brown et al. (2020).
In-context: We use the pretrained LM as it is and use in-context learning by conditioning on a concatenation of  $k$  training examples, following Brown et al. (2020).
PMI 0-shot, PMI In-context: We use the PMI method from Holtzman et al. (2021); Zhao et al. (2021) for 0-shot and In-context learning.
Channel 0-shot, Channel In-context: We use the noisy channel model from Min et al. (2022) for 0-shot and In-context learning.
Multi-task 0-shot: We train the LM on the same meta-training tasks without in-context learning objective, i.e., maximize  $P(y|x)$  without  $k$  other training examples, and then use zero-shot transfer on a target task. This is equivalent to MetaICL with  $k = 0$ . This is a typical multi-task learning approach from previous work (Khashabi et al., 2020; Zhong et al., 2021; Wei et al., 2022).
Channel Multi-task 0-shot: We have a channel variant of Multi-task 0-shot.
Fine-tune: We fine-tune the LM on an individual target task. This is not directly comparable to other methods as parameter updates are required for every target task.
Fine-tune w/ meta-train: We train the LM on meta-training tasks first and then further fine-tuned it on a target task. This is not directly comparable to other methods for the same reason as above.
Method	HR→LR	Class →Class	non-Class →Class	QA →QA	non-QA →QA	non-NLI →NLI	non-Para →Para
All target tasks
0-shot	34.8	34.2	34.2	40.2	40.2	25.5	34.2
PMI 0-shot	35.1	33.8	33.8	40.2	40.2	27.9	39.2
Channel 0-shot	36.5	37.3	37.3	38.7	38.7	33.9	39.5
In-context	38.2/35.3	37.4/33.9	37.4/33.9	40.1/38.7	40.1/38.7	34.0/28.3	33.7/33.1
PMI In-context	39.2/33.7	38.8/30.0	38.8/30.0	40.3/38.8	40.3/38.8	33.0/28.0	38.6/33.4
Channel In-context	43.1/38.5	46.3/40.3	46.3/40.3	40.8/38.1	40.8/38.1	39.9/34.8	45.4/40.9
Multi-task 0-shot	35.6	37.3	36.8	45.7	36.0	40.7	30.6
Channel Multi-task 0-shot	38.8	40.9	42.2	42.1	36.4	36.8	35.1
MetaICL	43.3/41.7	43.4/39.9	38.1/31.8	46.0/44.8	38.5/36.8	49.0/44.8	33.1/33.1
Channel MetaICL	49.1/46.8	50.7/48.0	50.6/48.1	44.9/43.5	41.9/40.5	54.6/51.9	52.2/50.3
Fine-tune	46.4/40.0	50.7/44.0	50.7/44.0	41.8/39.1	41.8/39.1	44.3/32.8	54.7/48.9
Fine-tune w/ meta-train	52.0/47.9	53.5/48.5	51.2/44.9	46.7/44.5	41.8/39.5	57.0/44.6	53.7/46.9
Target tasks in unseen domains
0-shot	32.6	32.6	32.6	45.9	45.9	33.4	38.3
PMI 0-shot	28.9	28.9	28.9	44.4	44.4	33.4	32.9
Channel 0-shot	29.1	29.1	29.1	41.6	41.6	33.1	32.6
In-context	30.6/27.5	30.6/27.5	30.6/27.5	45.6/44.7	45.6/44.7	52.0/41.3	35.8/34.1
PMI In-context	34.9/27.7	34.9/27.7	34.9/27.7	45.4/44.7	45.4/44.7	47.8/35.2	38.5/33.3
Channel In-context	39.6/33.6	39.6/33.6	39.6/33.6	44.7/40.6	44.7/40.6	40.4/35.7	44.1/36.8
Multi-task 0-shot	35.4	28.0	28.6	71.2	40.3	33.5	35.0
Channel Multi-task 0-shot	36.3	31.1	34.3	54.4	39.4	50.8	34.1
MetaICL	35.3/32.7	32.3/29.3	28.1/25.1	69.9/68.1	48.3/47.2	80.1/77.2	34.0/34.0
Channel MetaICL	47.7/44.7	41.9/37.8	48.0/45.2	57.9/56.6	47.2/45.0	62.0/57.3	51.0/49.9
Fine-tune	44.9/37.6	44.9/37.6	44.9/37.6	43.6/39.1	43.6/39.1	56.3/33.4	56.6/51.6
Fine-tune w/ meta-train	53.3/43.2	53.2/43.7	46.1/36.9	67.9/66.2	44.5/42.8	71.8/58.2	65.6/61.4
Table 5: Main results, using GPT-2 Large. Two numbers indicate the average and the worst-case performance over different seeds used for  $k$  target training examples. Bold indicates the best average result except results from fine-tuned models that are not comparable. 'Class' indicates 'Classification'.
We use Macro-F1 $^2$  and Accuracy as evaluation metrics for classification tasks and non-classification tasks, respectively.
For a target task, we use  $k = 16$  training examples, sampled uniformly at random. We relax the assumption of perfect balance between labels on  $k$  training examples, following Min et al. (2022). Because in-context learning is known to have high variance (Zhao et al., 2021; Perez et al., 2021; Lu et al., 2021), we use 5 different sets of  $k$  training examples. We first compute the average and the worst-case performance over seeds for every target task, and then report the macro-average of them over all target tasks.
4.4 Experiment Details
As a base LM, we use GPT-2 Large (Radford et al., 2019) which consists of 770M parameters. For baselines without meta-training (raw LMs), we also compare with GPT-J (Wang and Komatsuzaki,
2021), which is the largest public causal LM at the time of writing, consisting of 6B parameters.
Elimination of templates Prior work uses human-authored templates to transform the input-output pair to a natural language sentence (Zhong et al., 2021; Mishra et al., 2022; Wei et al., 2022; Chen et al., 2022). They require expensive manual effort (as 136 different templates are required for 136 tasks in this paper) and cause unstable model performance due to many different ways of writing (Mishra et al., 2021). We eliminate templates, using the given input (or a concatenation of inputs if there are multiple) and label words provided in the original datasets. A comparison of input-output schemes from prior work and our approach is shown in Table 4.
Training details All implementation is done in PyTorch (Paszke et al., 2019) and Transformers (Wolf et al., 2020). For meta-training, we use
4 In our preliminary experiments, we explored templates taken from prior work, but found that they do not consistently improve few-shot performance, even when they do improve zero-shot performance.
2More suitable than accuracy for imbalanced classification.
3Appendix C.2 reports performance for other LM sizes.
Method	HR→LR	Class →Class	non-Class →Class	QA →QA	non-QA →QA	non-NLI →NLI	non-Para →Para
All target tasks
Channel In-context	43.1/38.5	46.3/40.3	46.3/40.3	40.8/38.1	40.8/38.1	39.9/34.8	45.4/40.9
MetaICL	43.3/41.7	43.4/39.9	38.1/31.8	46.0/44.8	38.5/36.8	49.0/44.8	33.1/33.1
Channel MetaICL	49.1/46.8	50.7/48.0	50.6/48.1	44.9/43.5	42.1/40.8	54.6/51.9	52.2/50.3
GPT-J Channel In-context	48.6/44.4	51.5/47.0	51.5/47.0	47.0/45.2	47.0/45.2	47.2/41.7	51.0/47.5
Target tasks in unseen domains
Channel In-context	39.6/33.6	39.6/33.6	39.6/33.6	44.7/40.6	44.7/40.6	40.4/35.7	44.1/36.8
MetaICL	35.3/32.7	32.3/29.3	28.1/25.1	69.9/68.1	48.3/47.2	80.1/77.2	34.0/34.0
Channel MetaICL	47.7/44.7	41.9/37.8	48.0/45.2	57.9/56.6	47.2/45.0	62.0/57.3	51.0/49.9
GPT-J Channel In-context	42.8/38.4	42.8/38.4	42.8/38.4	55.7/54.4	55.7/54.4	51.1/40.4	52.0/46.5
Table 6: Comparison between raw LM in-context learning (based on GPT-2 Large and GPT-J) and MetaICL (based on GPT-2 Large). GPT-2 Large used unless otherwise specified. Two numbers indicate the average and the worst-case performance over different seeds used for  $k$  target training examples. For raw LM baselines, Channel In-context is reported because it is the best raw LM baseline overall across the settings; full results based on GPT-J are provided in Appendix C.1.
up to 16,384 training examples per task. We use a batch size of 8, learning rate of  $1 \times 10^{-5}$  and a sequence length of 1024. For multi-task 0-shot baselines (the baselines with no in-context learning), we use a sequence length of 256. We train the model for 30,000 steps. To save memory during metatraining, we use an 8-bit approximation (Dettmers et al., 2022) of an Adam optimizer (Kingma and Ba, 2015) and mixed precision (Micikevicius et al., 2017). Training was done for 4.5 hours with eight 32GB GPUs. This is drastically more efficient than recent prior work, e.g., 270 hours of a 512GB TPU in Sanh et al. (2022).
More details about preprocessing and training can be found in Appendix B.
Table 5 reports the full results using GPT-2 Large, where we compute the average and the worst-case performance of every target task and report the macro-average over them. The top and the bottom respectively evaluate on all target tasks and target tasks in unseen domains only.
Our baselines are strong We first discuss the results of ours baselines. Among raw LMs without meta-training (the first six rows of Table 5), we observe that channel in-context baselines are the most competitive, consistent with findings from Min et al. (2022). We then find that Multi-task 0-shot baselines do not outperform the best raw LM