SlowGuess/NAACL_2022_OCR huggingface dataset:
Mingda Chen $^{1*}$  Jingfei Du $^{2}$  Ramakanth Pasunuru $^{2}$  Todor Mihaylov $^{2}$  Srini Iyer $^{2}$  Veselin Stoyanov $^{2}$  Zornitsa Kozareva $^{2}$
<sup>1</sup>Toyota Technological Institute at Chicago, IL, USA
mchen@ttic.edu
{jingfeidu, rpasunuru, tbmihaylov, sviyer, ves, zori}@fb.com
Self-supervised pretraining has made few-shot learning possible for many NLP tasks. But the pretraining objectives are not typically adapted specifically for in-context few-shot learning. In this paper, we propose to use self-supervision in an intermediate training stage between pretraining and downstream few-shot usage with the goal to teach the model to perform in-context few shot learning. We propose and evaluate four self-supervised objectives on two benchmarks. We find that the intermediate self-supervision stage produces models that outperform strong baselines. Ablation study shows that several factors affect the downstream performance, such as the amount of training data and the diversity of the self-supervised objectives. Human-annotated cross-task supervision and self-supervision are complementary. Qualitative analysis suggests that the self-supervised-trained models are better at following task requirements.
In-context few-shot learning seeks to solve unseen tasks at inference time by conditioning on a few training examples. In particular, in this case we are interested in methods that forgo any weight updates (Brown et al., 2020). Prior work has been focused on improving inference time algorithms (e.g., rescoring generated outputs (Zhao et al., 2021), selecting (Liu et al., 2021) and ordering (Lu et al., 2021) the given few-shot examples) and incorporating extra resources (e.g., fine-tuning models on human-annotated datasets (Mishra et al., 2021; Ye et al., 2021; Wei et al., 2022)).
We hypothesise that a different way to improve in-context few-shot learning is through designing self-supervised objectives that more closely resemble the format of tasks that the model will be asked to perform. To do so, we cast the self-supervised
training as an intermediate training stage between language model pretraining and downstream few-shot evaluation. In particular, we construct training datasets based on the self-supervised objectives following similar formats used in the downstream tasks, fine-tune pretrained language model checkpoints on the training datasets, and then evaluate the models on benchmarks.
In experiments, we consider four self-supervised objectives, including masked word prediction and classification tasks related to next sentence prediction (Devlin et al., 2019). We evaluate models on two benchmarks (13 tasks in total): SuperGLUE (Wang et al., 2019) and Natural-Instructions (Mishra et al., 2021). SuperGLUE focuses on discriminative tasks, and Natural-Instructions is a set of generative tasks.
Empirically, we experiment with pretrained language models of two sizes: 125 million parameters and 1.3 billion parameters. We show that in our best setting, the 1.3 billion parameters model trained by the self-supervision performs better than the initial pretrained language models and two strong baselines on average.
Further analysis reveals that (1) the effectiveness of the self-supervision depends on the amount of training data, but the benefit of adding more data is diminishing; (2) the improvements brought by the self-supervision are in part due to the semantic similarity between the training and evaluation tasks; (3) adding more self-supervised objectives may not help model performance because adding them does not contribute to the diversity of the self-supervised tasks; (4) choosing similar task templates for both self-supervised and downstream tasks plays a vital role in improving model performance; (5) self-supervised tasks and human-annotated datasets are complementary; (6) generation examples show that compared to the initial pretrained language models, self-supervised-trained models are better at following the task instructions.
*Work done during an internship at Meta AI.
Human Language Technologies, pages 3558 - 3573
In-Context Few-Shot Learning. Brown et al. (2020) discover that large pretrained language models can solve unseen tasks at inference time. Recent work has improved the in-context few-shot performance by rescoring generated outputs (Zhao et al., 2021), selecting (Liu et al., 2021) and ordering (Lu et al., 2021) the given few-shot examples. Other work studies pretrained language models' cross-task generalization abilities for in-context few-shot or zero-shot learning using human-annotated datasets (Ye et al., 2021; Wei et al., 2022; Sanh et al., 2022; Min et al., 2021; Xu et al., 2022) via instructions (Weller et al., 2020; Efrat and Levy, 2020; Mishra et al., 2021; Ouyang et al., 2022) and retrieved examples (Hu et al., 2022; Lin et al., 2022). Our work differs in that we focus on self-supervised training.
Fine-Tuning for Few-Shot Learning. Pre-trained language models for few-shot learning typically follows the "pretrain then fine-tune" paradigm (Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019, inter alia), where recent work has focused on designing templates for few-shot fine-tuning (Reynolds and McDonell, 2021; Schick and Schütze, 2021a,c,b; Le Scao and Rush, 2021; Tam et al., 2021; Gao et al., 2021; Sorensen et al., 2022), and optimizing soft prompts (Li and Liang, 2021; Qin and Eisner, 2021; Lester et al., 2021; Gu et al., 2021; Zhang et al., 2022). Other work focuses on unifying task formats to maximize the benefits of human annotations, including question answering (Zhong et al., 2021), textual entailment (Yin et al., 2019, 2020; Wang et al., 2021a), and many other tasks (McCann et al., 2018; Keskar et al., 2019; Raffel et al., 2020; Bragg et al., 2021). In contrast, our focus is on in-context few-shot learning, without fine-tuning models on downstream task examples.
Pretraining for Few-Shot Learning. Several papers have adapted various resources for pretraining models to enhance their performances on few-shot learning, such as pretraining on hypertext (Aghajanyan et al., 2021b), question-infused pretraining (Jia et al., 2021), and self-training (Du et al., 2021; Vu et al., 2021; Wang et al., 2021b). Pretraining approaches have targeted specific tasks, such as task-oriented dialog (Mi et al., 2021), intent detection (Zhang et al., 2021), and data-to-text generation (Chen et al., 2020). Our work differs as we
use plain text as opposed to (naturally-occurring) human-annotated resources. Relatedly, Bansal et al. (2020) used self-supervised meta-learning for few-shot text classification rather than in-context few-shot learning.
Intermediate Fine-Tuning. Since our approach involves an extra training stage between pretraining and downstream evaluation, it is also related to prior work that uses multi-stage fine-tuning on human-annotated datasets for generic tasks (Phang et al., 2018; Pruksachatkun et al., 2020; Chang and Lu, 2021; Aghajanyan et al., 2021a; Poth et al., 2021) and text classification (Zhang and Zhang, 2021). Relevant work also studies intermediate fine-tuning using crosslingual supervision (Phang et al., 2020; Moghe et al., 2021). Rubino and Sumita (2020) use an intermediate self-supervised training stage for machine translation quality estimation.
We describe four self-supervised training objectives that will be used to train models before downstream evaluations.
We begin by defining the example and the instance used during our self-supervised training. An example is an input-output pair. To differentiate the input and the output, we append special tokens "Input:" and "Output:" to the beginning of input text and output text respectively where the two texts are also separated by the <newline> token (see Figure 1 for examples).<sup>1</sup>
An instance is a linearized string formed by several examples from the same task (e.g., see Figure 2). As we encode the text using causal attention, the examples closer to the beginning of input sequences can be seen as task demonstrations, resulting in efficient computation.
When constructing the training examples, we pick three or more consecutive sentences (depending on the minimum sequence length we enforce on the sentences) and then apply task-specific rules to automatically create training data. To form a training instance, we randomly select examples from the same task until reaching the maximum sequence length (i.e., 2048). During training, we compute a cross-entropy loss on tokens in the output texts.
<sup>1</sup>We chose this special symbol because we always start the self-supervised training from a pretrained language model checkpoint.
Original Raw Text
Natural language processing is a subfield of computer science concerned with the interactions between computers and human language. The goal is a computer capable of "understanding" the contents of documents.
Next Sentence Generation
Input: Natural language processing is a subfield of computer science concerned with the interactions between computers and human language. Output: The goal is a computer capable of "understanding" the contents of documents.
Input: Natural language processing is a subfield of computer science concerned with the interactions between computers and human language. The following is a list of some of the most commonly researched tasks in computer vision.
Output: False
Last Phrase Prediction (Generation)
Input: Natural language processing is a subfield of computer science concerned with the interactions between computers and human language. Question: The goal is a computer capable of "understanding"?
Output: the contents of documents.
Masked Word Prediction
Input: Natural language processing is a subfield of computer science concerned with the interactions between _____. The goal is a computer capable of "understanding" the contents of documents.
Output: computers and human language
Last Phrase Prediction (Classification)
Input: Natural language processing is a subfield of computer science concerned with the interactions between computers and human language. Question: The goal is a computer capable of "understanding"? Answer: the development of new models.
Figure 1: Examples of our self-supervised training tasks. Each example is an input-output pair constructed from the raw text.
Input: Natural language processing is ...<newline> Output: ... the contents of documents. <newline> Input: Computer vision deals with ...<newline> Output: ... visual system. <newline>
Figure 2: An example of a training instance. Each instance is formed by several training examples. During training, we use left-to-right language models and compute a cross-entropy loss on the output texts (indicated by the red color in the shown example). We note that when computing the loss on the second example, the first example can be seen as task demonstrations. For brevity, we show part of the input and output texts.
We describe details of the self-supervised tasks in the following subsections.
3.1 Next Sentence Generation
In light of the strong performance of language models on in-context few-shot learning (Brown et al., 2020), we incorporate the language modeling as one of our self-supervised tasks, which we call "next sentence generation" (NSG). NSG asks the model to generate the next sentence given previous sentences as context. When building data for this task, we use the last sentence as output and the rest of the sentences as input.
3.2 Masked Word Prediction
The second task we consider is based on masked word prediction (MWP) which is commonly used in pretraining generic text encoders (Devlin et al., 2019; Liu et al., 2019). The task asks the model to fill in the missing information based on the surrounding context. Specifically, MWP randomly replaces words in input sentences with a special symbol and requires models to recover the masked words in the input. For this task, we create input text by randomly replacing  $1\sim 20$  words in the input text with a special token $^{2}$  and use the masked out
words as the output text.
3.3 Last Phrase Prediction
Inspired by the LAMBADA dataset (Paperno et al., 2016), a question answering dataset which asks models to predict the last word in a sentence given several sentences of context, we create a "last phrase prediction" (LPP) task, which requires predicting the last phrase in a sentence. To solve this task, models need to draw relevant information from the context and the learned knowledge during pretraining. We cast LPP as either a generation task or a classification task. The latter variant of LPP is a binary classification task that labels if the given answer is the correct phrase. To facilitate a unified format of these two tasks, we append a special token "Question:" to the beginning of the last sentence and replace the last phrase with a question mark. For the classification LPP, we separate the given answer and the previous context and sentences with a special token "Answer:". An example of this task is shown in Figure 1.
More specifically, we identify the last phrase of a sentence based on a set of function words (see appendix A.5 for the list of function words). If there are multiple function words in a sentence, we pick the last one. Then we treat the text segment starting from the function word as the last phrase.<sup>3</sup>
3We ensure that the last sentence in raw text for this task always has at least one valid function word and the function
$^{2}$ We randomly select the special token from the following list: _, ⟨⟩, @@@, (), $$$, %%%, ###, ***, and +++. We use random symbols instead of a fixed symbol because we found that it gives better performance in our preliminary experiments.
Figure 3: Example illustrating the construction of training instances for our classification task. There are four input types, and each training instance has two or three types. As the shown instance has the following two types: "original sentences" and "different documents", it comprises examples from two different documents. The instance resembles the next sentence prediction task, encouraging models to compare topical similarities between the two examples.
When selecting negative answers, we randomly choose from the phrases extracted from the same function words (to make the negative answers more challenging).
3.4 Classification
Similar to the next sentence prediction task (Devlin et al., 2019) and the sentence ordering prediction task (Jernite et al., 2017; Chen et al., 2019) for pretraining language representations, we create a classification task (CL) for our self-supervised training. As shown in Figure 3, for this task, we consider four types of input: original sentences, shuffled sentences, sentences from a different document, and sentences from multiple documents. In particular, for original sentences, we directly use text from original human-written documents. For shuffled sentences, we randomly shuffle all the input sentences. For sentences from multiple documents, we randomly replace  $50\%$  of the input sentences with sentences from another document. We also ensure that the selected sentences (from both the input and another document) are consecutive in their original documents. For sentences from different documents, we replace the input sentences with sentences from another document. See Figure 3 for an example of each type of input.
When constructing a training instance, we randomly pick one or two additional input types and combine them with the original sentences to form a binary or three-way classification task. We also randomly assign label strings to input types in each instance to ensure that models follow the information given by earlier examples when making predictions.
The classification task is different from the other self-supervised tasks described in earlier subsec
word lies at the second half of the sentence.
tions. It explicitly requires models to compare inputs across examples in a training instance to determine if the given input shares similar properties with the others.
For the pretrained language model checkpoints, we use the 125 million parameters (125M) and the 1.3 billion parameters (1.3B) dense model from Artetxe et al. (2021). These pretrained models have shown results comparable to GPT3 across various tasks.
For self-supervised training, we use a subset of documents from the RoBERTa training corpus (Liu et al., 2019) that contains four domains: BOOK-CORPUS plus Wikipedia, CC-NEWS, OPENWEB-TEXT, and STORIES. Specifically, we randomly sample 100k documents from each domain except STORIES where we only sample 10k documents as the documents there are much longer than the others. The final training data contains approximately 1 million instances with 250k training instances per task. For the 125M model, we train for 10 epochs, which takes roughly 1 day on a V100 GPU. For the 1.3B model, we train for 5 epochs, which takes roughly 3 days on 2 V100 GPUs.
4.2 Evaluation Setup
The instance and example during evaluation shares similar definition as those in Sec. 3 except that each evaluation instance has only one example from test splits and it is placed at the last position in the instance. The other examples in the instance
4The average numbers of example per instance for each data source are: 6.9 for BOOKCORPUS plus Wikipedia, 5.3 for CC-NEWS, 3.5 for OPENWEBTEXT, and 7.2 for STORIES.
GPT3	${Context} {newline} ${Question} {newline} - [{${Label}]} ${Answer}
Ours	Input: ${Context} Question: ${Question} Answer: ${Answer} {newline} Output: ${Label}
Table 1: Evaluation templates for MultiRC.  $\mathbb{S}\{\cdot \}$  represents values drawn from a particular data field. We alter the GPT3 template for this task to share a similar format with one of our self-supervised tasks (i.e., classification LLP in this case). The red, boldfaced texts are used to compute the language modeling perplexities for ranking the labels. We note that the shown template is for a single example, and there could be multiple examples within an instance.
(i.e., task demonstrations) come from either training splits or task-specific instructions depending on benchmarks.
We evaluate the models on two benchmarks: SuperGLUE and Natural-Instructions. SuperGLUE is a set of tasks focusing on natural language understanding. We use BoolQ (BQ; Clark et al., 2019), CB (De Marneffe et al., 2019), COPA (CA; Roemmele et al., 2011), MultiRC (MC; Khashabi et al., 2018), and RTE (RE; Giampiccolo et al., 2007; Bentivogli et al., 2009; Dagan et al., 2006; Bar Haim et al., 2006). We report results for the official development sets. The task demonstrations are examples randomly selected from the training sets. We report mean and standard deviations of five runs with different random seeds. Following GPT3, we use a ranking based approach when evaluating the models (i.e., pick the best label based on language modeling perplexities).
Natural-Instructions. Natural-Instructions evaluate models' cross-task generalization abilities where all the tasks are generation tasks. It splits the tasks into two groups for training and evaluation. We use the same task split and evaluate models on the following task categories: question generation (QG), answer generation (AG), minimal modification (MM), and verification (VF). Each task category has two tasks. Following the few-shot setting used in Mishra et al. (2021), we evaluate models using 100 examples per task, use greedy decoding, and report ROUGE-L (Lin, 2004) scores per task category. For task demonstrations, we use the positive examples in the instructions in
5We exclude WSC (Levesque et al., 2011) and ReCoRD (Zhang et al., 2018) as pretrained models, including GPT3, require scoring algorithms at inference time to achieve competitive results. We exclude WiC (Pilehvar and Camacho-Collados, 2019) because GPT3-like models, including GPT3 and our models, do not give accuracies significantly better than random baselines.
<sup>6</sup>We discard training tasks that share the same source datasets with evaluation tasks as we found that tasks with the same source dataset may contain leaked labels. We exclude the binary classification tasks because the class labels are severely imbalanced (i.e., more than  $80\%$  of the class labels belong to one category).
Natural-Instructions.
SuperGLUE. As our self-supervised tasks are formatted as input-output pairs, we change the task-specific templates for SuperGLUE to make them more similar to our self-supervised tasks. For example, as shown in Table 1, we make MultiRC similar to the classification LPP. More details of the template changes are in appendix A.6.
For both benchmarks, we also report an averaged performance for each model. For SuperGLUE, the average performance is computed based on the means of task performances. When a task has two metrics, we take the average of the two as the task performance.
More details on the dataset statistics and metrics for each task for both benchmarks are in appendix A.2.
Baselines. We consider four baselines: (1) directly evaluating pretrained language models on the benchmarks (LM); (2) performing additional language modeling training on the subset of the original data that is used for constructing the self-supervised tasks (ExtraLM). We use ExtraLM to approximately measure the contribution of additional computation; (3) fine-tuning on training sets for the tasks outside the evaluation sets (CrossTask). We use CrossTask to estimate the performances of cross-task supervision from human-annotated datasets; and (4) fine-tuning on training sets for the tasks in the evaluation sets (SameTask). SameTask serves as an oracle baseline estimating the approximated upperbound performances of cross-task supervision.
Since SuperGLUE does not have an official split for the CrossTask setting, we split the datasets into two groups according to the task category and report the CrossTask results based on "CrossTask  $(\mathrm{QA}\rightarrow \mathrm{NLI})$  and "CrossTask  $(\mathrm{NLI}\rightarrow \mathrm{QA})$ ". As we alter the task templates, we report results for evaluating the pretrained language model check
${}^{7}$  "QA  $\rightarrow$  NLI" suggests that we train models on the NLI tasks and evaluate on the QA tasks. Similarly,for "NLI  $\rightarrow$  QA", we train models on the QA tasks and evaluate on the NLI tasks.
Model	MS	BoolQ	MultiRC	COPA	RTE	CB	Avg.
LM	125M	52.1(1.7)	5.2(0.7)/49.5(1.1)	67.6(2.3)	52.0(1.2)	50.7(3.2)/34.8(2.5)	48.4
ExtraLM	125M	51.5(1.7)	5.1(0.8)/49.7(1.0)	68.0(1.6)	52.3(1.2)	49.5(4.6)/35.5(5.6)	48.3
NewTemplate	125M	52.2(1.8)	5.2(0.6)/47.9(1.4)	63.0(2.5)	50.8(2.0)	46.4(7.3)/30.1(6.4)	46.2
CrossTask(NLI→QA)	125M	38.1(0.3)	5.1(0.7)/43.5(2.5)	65.4(2.1)	-	-	42.2
CrossTask (QA→NLI)	125M	-	-	-	53.6(0.5)	39.6(1.5)/19.9(1.2)
SameTask	125M	71.2	19.9/66.9	72.0	67.3	71.4/60.2	61.9
Self-Supervised	125M	55.7(0.6)	7.0(1.0)/60.2(0.3)	67.6(2.1)	53.0(1.5)	50.0(5.2)/39.8(3.0)	51.0
LM	1.3B	48.6(2.3)	5.5(0.5)/53.7(0.7)	83.4(1.7)	51.9(1.2)	53.6(5.2)/37.2(3.7)	51.8
ExtraLM	1.3B	49.6(1.9)	4.9(0.6)/54.8(0.6)	82.6(1.5)	52.9(1.9)	51.4(7.5)/35.6(5.3)	51.7
NewTemplate	1.3B	51.3(1.3)	5.0(0.4)/52.8(1.2)	81.2(2.4)	50.8(2.3)	49.3(4.7)/33.7(4.2)	50.7
CrossTask(NLI→QA)	1.3B	53.4(0.8)	1.2(0.3)/57.2(0.3)	76.2(2.9)	-	-	49.6
CrossTask (QA→NLI)	1.3B	-	-	-	54.3(1.2)	44.6(3.6)/25.2(4.9)
SameTask	1.3B	77.1	27.5/71.6	85.0	68.1	75.2/64.3	69.9
Self-Supervised	1.3B	61.7(0.3)	5.2(0.1)/62.1(0.3)	84.0(2.7)	53.1(0.7)	54.3(2.0)/37.0(1.9)	55.6
Table 2: SuperGLUE results. We report mean and standard deviations (the numbers in parenthesis) of five runs. The best result (we take the average if there are two metrics) except SameTask in each column for each model size is boldfaced. MS=model size.
Model	MS	QG	AG	MM	VF	Avg.
GPT3	-	43.0	50.0	70.0	32.0	48.8
LM	125M	33.7	12.9	53.0	14.7	28.6
ExtraLM	125M	34.4	13.4	53.7	14.3	28.9
CrossTask	125M	22.0	24.8	66.9	17.9	32.9
SameTask	125M	54.8	42.3	77.3	78.3	63.2
SelfSup.	125M	16.9	14.6	70.1	18.9	30.0
LM	1.3B	40.9	32.5	74.0	27.8	43.8
ExtraLM	1.3B	41.1	32.7	75.9	25.2	43.7
CrossTask	1.3B	38.1	41.6	69.2	23.0	42.9
SameTask	1.3B	55.5	64.6	81.0	80.4	70.4
SelfSup.	1.3B	43.9	37.5	72.3	28.6	45.5