Pre-Training to Learn in Context
Year: 2023

Abstract:
In-context learning, where pre-trained language models learn to perform tasks from task examples and instructions in their contexts, has attracted much attention in the NLP community. However, the ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. To this end, we propose PICL (Pretraining for In-Context Learning), a framework to enhance the language models’ in-context learning ability by pre-training the model on a large collection of “intrinsic tasks” in the general plain-text corpus using the simple language modeling objective. PICL encourages the model to infer and perform tasks by conditioning on the contexts while maintaining task generalization of pre-trained models. We evaluate the in-context learning performance of the model trained with PICL on seven widelyused text classification datasets and the SUPERNATURALINSTRCTIONS benchmark, which contains 100+ NLP tasks formulated to text generation. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters. The code is publicly available at https://github. com/thu-coai/PICL.

heading: 1 Introduction
text: Pre-trained language models (PLMs; Han et al., 2021; Qiu et al., 2020) have shown strong abilities of learning and performing unseen tasks conditioning on several task examples or instructions in its context, which is called in-context learning (ICL; Brown et al., 2020). Compared to conventional fine-tuning methods, ICL adapts PLMs to downstream tasks only through inference, without parameter updates, which is computationally cheaper in practice and is closer to general AI.
† Corresponding author. ∗ Contribution during internship at Microsoft Research.
However, PLMs trained on massive corpora to predict the next word given previous words are not explicitly taught to learn in the context. This makes ICL a surprising emergent ability but also indicates that the ICL ability of PLMs is not fully exploited. Garg et al. (2022) has shown that by directly training to do ICL in a meta-learning paradigm, models show strong performance on learning simple function classes in the context. In practical NLP scenarios, previous works (Min et al., 2022b; Chen et al., 2022b) also enhance the ICL performance by metafine-tuning PLMs on a large collection of downstream tasks and evaluating them on unseen tasks. However, the low diversity of human-annotated downstream tasks restricts the performance of the meta-tuned model. Direct training on downstream tasks also brings undesired bias on specific input formats, label spaces, or domains, which hurts the generalization of PLMs.
To enhance the ICL ability while maintaining generalization, we propose PICL (Pre-training for In-Context Learning), a framework that exploits the PLM’s ICL ability by pre-training models on data automatically constructed from the general plain-text corpus. Our framework is based on a simple observation that many paragraphs in the text documents contain “intrinsic tasks”. As shown in the left part of Figure 1, each paragraph in the document contains an intrinsic task. When doing language modeling on each paragraph, models implicitly perform the corresponding intrinsic tasks simultaneously. This shares a similar idea with the prompt-learning paradigm (Liu et al., 2021), where downstream data examples from NLP tasks are transformed into text sequences, and the model learns to perform the original tasks when trained on the text sequences with language modeling. Different from the downstream data, text paragraphs contain more diverse intrinsic tasks and have little bias on input formats, label spaces, or domains because they are free-form texts from the large-
4849
scale general corpus. By gathering and concatenating paragraphs with the same intrinsic tasks (right part of Figure 1), we can construct a meta-training dataset to pre-train the model to perform the intrinsic task conditioning on paragraphs in the context, and thereby improve the ICL ability.
We adopt a retrieval-based approach to gather paragraphs sharing the same intrinsic tasks from a general corpus. We first train an encoder to represent each paragraph in a vector space where paragraphs with the same intrinsic task have close embeddings. The encoder is trained with contrastive learning (Khosla et al., 2020) on a collection of downstream datasets by taking examples from the same tasks as positive pairs and those from different tasks as negative pairs. Then, treating any paragraph in the corpus as a query, we retrieve the paragraphs with close representations to the query, namely, sharing the same intrinsic task with the query. Finally, we concatenate the query and the retrieved paragraphs to get a pre-training instance. Note that although we use downstream datasets, the model is trained on instances constructed from the general corpus, which ensures its generalization.
We evaluate the ICL performance of the model pre-trained with PICL on seven widelyused text classification datasets and SUPERNATURALINSTRUCTIONS (Wang et al., 2022), a benchmark whose test split contains more than 100 tasks formulated into text generation. Empirical results show the effectiveness of PICL, enabling the model to reach or even outperform larger models with nearly 4x parameters. Besides, we find that the PICL-trained model is more generalizable on
various tasks than previous meta-fine-tuning methods. We also conduct extensive experiments to analyze several key factors of PICL.

heading: 2 Method
text: We first present an overview of PICL and then describe the details in the following sections. As shown in the right part of Figure 1, we construct the pre-training instances from a corpus C consisting of paragraphs split from full documents by “\n”. For each paragraph z0 in C, we first use a retriever R to find k paragraphs {z1, z2, · · · , zk} sharing the same intrinsic task (Sentiment Analysis) with z0. Then the retrieved paragraphs are treated as demonstrations and concatenated with z0 to form a pre-training instance: zk ⊕ zk−1 ⊕ · · · ⊕ z1 ⊕ z0. Finally, we adopt a language modeling objective to pre-train the model on the constructed instances.
In this way, the pre-training stage can be regarded as a meta-training process, where the model learns to solve the intrinsic task in z0 conditioning on its context zk ⊕ zk−1 ⊕ · · · ⊕ z1. Since C is a large-scale general corpus, it contains a variety of intrinsic tasks and little domain bias, which ensures the generalization of the pre-trained model.

heading: 2.1 Retriever
text: The main component of the retriever R is a tasksemantics encoder E that represents a text paragraph as a d-dimensional vector in a space V , where paragraphs with the same intrinsic tasks have similar representations. We define the similarity between two paragraphs z0 and z using the dot product of their representations: E(z0) · E(z).
Encoder We use RoBERTaBASE (Liu et al., 2019) as the base model of E. The output vector is computed by averaging the last-layer representation of each token in the input paragraph.
Retrieval We approximate that paragraphs whose representations are close to each other in V share the same intrinsic task. Therefore, for every paragraph z0 in C, R searches for k paragraphs with embeddings closest to E(z0):
R(z0) = {zk, zk−1, · · · z1} = top-kz(E(z0) · E(z)). (1)
We employ the FAISS library (Johnson et al., 2019) for efficient searching.
Contrastive Training We adopt contrastive learning (Khosla et al., 2020; Karpukhin et al., 2020) to train the task-semantics encoder E. As shown in Figure 2, we take two paragraphs with the same intrinsic task as positive pairs and those from different tasks as negative pairs. However, the annotation of a paragraph’s intrinsic task is usually unavailable. To this end, we use a collection of downstream NLP datasets from various tasks whose examples are converted into text sequences with human-written prompts to train E. In this way, treating each text sequence as a paragraph, we can regard the corresponding downstream task as the intrinsic task annotation. We assume that the instances from all downstream tasks form a dataset D. For each z0 ∈ D, we have a positive instance z+ sharing the same task with z0 and a set N (z0) consisting of negative instances with different tasks
than z0, the loss function takes the form:
L(z0, z+,N (z0))
= − log e E(z0)·E(z+)
eE(z0)·E(z+) + ∑
z−∈N (z0) eE(z0)·E(z−)
. (2)
Positive and Negative Instances For each z0 ∈ D, we randomly sample a positive instance z+ belonging to the same task with z0 from D\{z0}. As shown in Figure 2, N (z0) contains two kinds of negative instances: (1) Easy Negatives z−easy sampled from D and belonging to different tasks than z0. (2) Hard Negatives z+hard sharing the same prompt with z0 but containing mismatched tasks. For instance, in Figure 2, we apply the prompt from the sentiment task to the summarization task to create the hard negative instance z−hard. This prevents the model from hacking the contrastive objective using prompts like “Guess the sentiment” and learning a trivial pattern matching but forces the model to extract task semantics from the whole paragraph.

heading: 2.2 Data Construction
text: For each paragraph z0 ∈ C, we concatenate the retrieved paragraphs {z1, z2, · · · , zk} = R(z0) with z0 to get a pre-training instance zk ⊕ zk−1 ⊕ · · · ⊕ z1 ⊕ z0. To improve the quality of the constructed data, we derive an approach to filter out instances that are less informative to ICL. We consider the following score to measure the informativeness of an instance based on the perplexity difference of the paragraphs in the instance before and after they are concatenated as a sequence:
s = −∑ki=0 logP (zi) + logP (zk ⊕ zk−1 ⊕ · · · ⊕ z0)
|zk ⊕ zk−1 ⊕ · · · ⊕ z0| ,
(3)
where | · | is the length of a sequence and P (·) is the language modeling probability based on any uni-direct PLMs. Given a manually set threshold δ, we retain the instances that satisfy s > δ. This criterion leverages the original ICL ability of the PLM. If concatenating the paragraphs results in lower perplexity, they are more correlated and may be more informative for ICL. We finally construct a pre-training corpus containing N instances Cpre-train = {zik ⊕ zik−1⊕, · · · ,⊕zi1 ⊕ zi0}Ni=1.

heading: 2.3 Pre-Training
text: We pre-train the model with auto-regressive language modeling on Cpre-train. Unlike previous works (Min et al., 2022b; Chen et al., 2022b), which only compute the language modeling loss on
the label tokens, we compute the loss on the whole sequence. There are two reasons for this choice. First, the intrinsic tasks are already in the natural language format, and it is unnecessary to split the input and the label. Second, we argue that computing loss on the whole sequence ensures a large token number in a forward batch, which is critical to maintaining the basic in-weights ability (Chan et al., 2022). Therefore, the loss function is:
LICL(θ) = − 1 N
N∑
i=1
logP (zik ⊕ zik−1 ⊕ · · · ⊕ zi0; θ), (4)
where θ is the parameters of the model. In addition, we find that adding a language modeling loss LLM(θ) on the original full documents before being split into paragraphs benefits the performance. Therefore, the final optimization objective is:
min θ
αLICL(θ) + (1− α)LLM(θ). (5)
where we set α = 0.5 in our main experiments.

heading: 3 Experimental Setup

heading: 3.1 Pre-training Data
text: We merge OPENWEBTEXT (Gokaslan et al., 2019), WIKICORPUS (Foundation, 2022), and BOOKCORPUS (Zhu et al., 2015) to construct the pre-training data, where full documents are split into paragraphs by “\n”. The corpus C consists of 80M paragraphs, totaling about 30GB. For each paragraph, we search for k = 20 demonstrations and concatenate them until 1024 tokens, the maximum input length constraint of the language model we used. This ensures that the model sees various demonstration numbers during pre-training. We use GPT2-Large (Radford et al., 2019) to compute P (·) in Equation 3 and set δ = 0.0 for filtering. More details of data processing and statistics are shown in Appendix A.

heading: 3.2 Baselines
text: We consider four baselines in our experiments:
• VanillaICL directly prompts a PLM with the concatenation of training examples to do ICL. • ExtraLM further pre-trains the PLM on the original full documents before being split into paragraphs with the language modeling objective. • Self-Sup (Chen et al., 2022a) designs four selfsupervised pre-training objectives, including Next Sentence Generation, Masked Word Prediction, Last Phrase Prediction, and Classification,
to enhance the ICL performance. We conduct the self-supervised pre-training on our merged corpus for a fair comparison. • MetaICL (Min et al., 2022b) meta-trains the model on a large collection of downstream human-annotated datasets for learning to learn in context. The meta-training instances are constructed by concatenating several training examples in each dataset to a single text sequence. We replicate the method on the training set of our task-semantics encoder for a fair comparison.

heading: 3.3 Evaluation
text: We evaluate the model trained with PICL on two kinds of downstream tasks.
Few-Shot Text Classification We consider seven widely-used text classification datasets, including SST-2 (Socher et al., 2013), SST-5 (Socher et al., 2013), Subj (Pang and Lee, 2004), MR (Pang and Lee, 2005), RTE (Dagan et al., 2006), CB (De Marneffe et al., 2019), and AGNews (Zhang et al., 2015) to evaluate the few-shot ICL performance of the trained models (see Appendix B.1 for more details). Note that these tasks are not included in the training set of the tasksemantics encoder. We randomly sample 4 or 8 demonstrations from the official training sets of each dataset. Effects of other demonstration numbers can be found in Section 4.3. We compute the average accuracy scores on at most 1000 samples from the validation split of each dataset across five random seeds for selecting demonstrations.
Instruction Following To test the generalization of PICL, we also evaluate the trained model on a larger range of tasks with more free-form inputs, including both human instructions and fewshot examples. We use the test split of SUPERNATURALINSTRUCTIONS (Wang et al., 2022) as the benchmark and exclude the tasks that appear in the training set of the task-semantics encoder, resulting in 105 evaluation tasks (see Appendix B.2 for a full list of tasks). Each task is specified with a human-written instruction and two or three demonstrations. We follow Wang et al. (2022) to formulate all tasks to the text generation format and score the outputs with ROUGE-L (Lin, 2004).

heading: 3.4 Settings
text: Retriever We train the task-semantics encoder on 37 tasks (see Appendix C) using up to 10K examples per task. To enhance generalization, we
apply multiple prompts from PromptSource (Bach et al., 2022) to one example and use 320 prompts in all. We use the in-batch negative trick (Chen et al., 2020) to compute the contrastive loss. We set the learning rate to 5× 10−5, the batch size to 64, and construct 4 hard negatives for each instance. The encoder is trained from RoBERTaBase for 1 epoch.
Language Model We test PICL based on the 770M GPT2-Large (Radford et al., 2019) unless otherwise specified. Results on larger models can be found in Appendix E.1. To save computational resources, we train the model from its pretrained checkpoints. We also test the VanillaICL performance of larger models, including GPT2xLarge (Radford et al., 2019) (1.5B) and GPTNeo (Black et al., 2021) (2.7B) for reference.
Pre-Training We set the maximum learning rate to 1×10−6 and use the “inverse square root” scheduler (Vaswani et al., 2017) with 1000 steps warmup. The model sees 131K tokens in a step and is pretrained for 100K steps. It takes less than a day to finish pre-training on 64 V100 32G GPUs.

heading: 4 Results

heading: 4.1 Few-Shot Text Classification
text: Table 1 shows the results of few-shot text classification, from which we have 3 observations.
First, among the baselines with 770M parameters, simply further training the model on our corpus with language modeling improves the performance (ExtraLM). This is likely due to the higher
domain diversity of our corpus. MetaICL is helpful on most datasets, which verifies the effectiveness of meta-training for ICL. Self-Sup fails to bring benefits on most datasets against VanillaICL, probably because the constrained label space of the Classification training task (only contains “True” and “False”) brings bias to the model’s output. This emphasizes the importance of using training objectives with little bias.
Second, we observe that the PICL-trained model outperforms the baselines with the same model sizes by a large margin on most datasets across different shots, verifying the effectiveness of PICL. An exception is RTE, where MetaICL performs the best. We speculate the reason is that some training tasks of MetaICL share the same label space with RTE (“Yes”/“No”), such as paraphrase identification. Min et al. (2022c) has shown that the label space plays a vital role in ICL, which explains the good performance of MetaICL on RTE.
Thrid, comparing models across different sizes, we find that increasing the model parameters boosts the performance, but PICL enables the 770M model to beat a 2.7B counterpart. This indicates that the ICL ability can be enhanced not only through scaling up the parameters. Improving the structure of the pre-training data is also beneficial. In Appendix E.1, we can see that PICL is also effective when applied to a 1.5B model.

heading: 4.2 Instruction Following
text: The results on SUPER-NATURALINSTRUCTIONS are shown in Table 2. We can see that PICL
achieves higher overall instruction following performance than the baselines, outperforming a larger model with about 4x parameters.
In Figure 3, we compare the per-task performance of PICL and MetaICL because they share the most similar setting where human-annotated downstream datasets are used. We observe that PICL outperforms MetaICL on about 3/4 of evaluation tasks, indicating that compared to fine-tuning directly on downstream tasks, pre-training on intrinsic tasks constructed from the general plain-text corpus brings better ICL ability and ensures higher generalization performance across a broad range of tasks (see Appendix E.2 for more details).
Most tasks where MetaICL beats PICL belong to text classification whose output spaces are “Yes/No” or “True/False”. This matches the second observation in Section 4.1, where MetaICL predicts “Yes/No” well because of training on tasks that share the same label spaces. On the other hand, PICL performs much better on text generation, or tasks whose output spaces share the same semantics with “Yes/No” but use label words not in the training tasks of MetaICL (e.g., “Correct/Wrong”). This indicates that direct training on downstream datasets causes overfitting to specific labels. There are also tasks where PICL performs similarly to MetaICL, such as reasoning and word analogy. We
notice that the improvements of PICL and MetaICL on these tasks are also marginal against VanillaICL probably because these tasks rely more on the “inweights learning” ability (Chan et al., 2022), rather than in-context learning.

heading: 4.3 Analysis
text: Effect of Retriever We compare different approaches to retrieve paragraphs and test the final model performance. We try randomly selecting paragraphs (Random), retrieving using the nonparametric approach (BM25), encoding each paragraph with the original pre-trained encoder as it is (RoBERTa), or using the encoder for sentence similarity (Reimers and Gurevych, 2019) (SRoBERTa). We also study different numbers of hard negatives (0, 1, 4) and downstream tasks (7, 24, 37) to train the task-semantics encoder in PICL. From the results in Table 3, we can see that all retrieval methods except Random bring improvements against VanillaICL on both text classification and instruction following settings, indicating that improving the coherence of the paragraphs in the pre-training data benefits ICL. Using the task-semantics encoder in PICL achieves the best performance, showing the importance of retrieving paragraphs based on task semantics rather than word overlap or sentence meanings. Comparing different settings to train the task-semantics encoder, we observe that increasing the number of hard negatives and training tasks improves the final performance. This is in line with previous works (Karpukhin et al., 2020; Chen et al., 2020; He et al., 2020) that more challenging hard negatives benefit contrastive learning.
Effect of Demonstration Numbers Training with PICL brings two benefits: (1) PLMs learn a format where demonstrations from the same task are concatenated as the prefix, which is beneficial when the model is evaluated under the same number of demonstrations. (2) PLMs learn a better
ability to infer and perform tasks from the context, even when the demonstration numbers in evaluation and pre-training do not match. To differentiate these effects, we conduct pre-training on instances containing only 4, 8, or 16 demonstrations and test the trained models under different text classification shots. Results in Figure 4 show that when pre-trained with different demonstration numbers, the models generalize well to unseen demonstration numbers in evaluation, achieving similar performance with the default setup where the model sees various demonstration numbers in pre-training (PICL-default). This indicates that the models learn more than the input formats in PICL.
Effect of Filtering We try different threshold values δ for filtering and report the scores on text classification tasks in Figure 5(a), while controlling the sizes of the constructed pre-training data the same. We find that δ = 0 yields the best performance,
which means we retain an instance if and only if the perplexity of individual paragraphs is higher than that of the concatenated sequence (Equation 3). For lower δ, the pre-training data contain too many uninformative instances for ICL. For larger δ, we speculate that the filtering process relies on the original GPT2-Large too much. Since we also pretrain based on GPT2-Large, the filtering process reduces the signals in the constructed data beyond the base model’s ability.
Effect of Full Documents In Figure 5(b), we report the model performance on text classification tasks when using different choices of α, which controls the proportion of the full-document data. We find that balancing the constructed and fulldocument data performs the best (α = 0.5). When α is too large, the model is trained mostly on our constructed data and overfits its bias inevitably introduced by the task-semantics encoder in the data construction process. When α is too small, our method degenerates into ExtraLM.
Effect of Data Amount We study the size effect of the corpus used to construct the pre-training data in PICL and report the performance on text classification tasks in Figure 6(a). We conduct the data construction on 0.01%, 0.1%, 1%, and 10% of the original 80M paragraphs (100%) and pre-train models for at most 100K steps until the validation
losses begin to increase. From the results, we conclude that when the corpus is small, pre-training with the constructed data hurts the performance because the search library is too small to find paragraphs sharing the same intrinsic tasks. Training on small data for multiple epochs also causes overfitting. When the corpus contains more than 80K paragraphs (0.1%), adding more data constantly improves the performance, which is consistent with the scaling law (Kaplan et al., 2020).
Data Comparison We compare the usefulness of different pre-training data to enhance the ICL ability. In addition to the final model performance, we borrow the thoughts for designing the filtering criterion in Section 2.2 to measure the usefulness of a pre-training instance by computing the perplexity using a reference large PLM: GPT-J (Wang and Komatsuzaki, 2021) with 6B parameters. Lower perplexity means the correlation within the instance is higher and is intuitively more helpful for enhancing the ICL ability. In Figure 6(b), we show the perplexity and the final model performance of 3 pretraining data: original full documents before being split into paragraphs (Full Doc), concatenation of randomly selected paragraphs (Rand ⊕ z0), and the concatenated same-intrinsic-task paragraphs gathered using the retrieval method in PICL before filtering (R(z0) ⊕ z0). We can see that the
data constructed by retrieval has much lower perplexity and correspondingly yields higher accuracy scores, which verifies its usefulness. In Appendix F, we present several examples of the retrieved paragraphs and the corresponding intrinsic tasks.

heading: 5 Related Work
text: In-Context Learning Recently, in-context learning (ICL), where models perform tasks simply conditioning on instructions or the concatenation of examples in the context (Brown et al., 2020), has been found promising for using PLMs in various application scenarios. To this end, there emerge many works to improve the ICL performance by calibrating the model predictions (Zhao et al., 2021; Han et al., 2022; Holtzman et al., 2021; Min et al., 2022a), selecting or reordering demonstrations (Rubin et al., 2022; Liu et al., 2022; Lu et al., 2022), designing pre-training tasks (Chen et al., 2022a), and breaking the context length limits (Hao et al., 2022). However, the underlying mechanism of ICL is poorly understood (Min et al., 2022c). Therefore, some works propose mathematical frameworks to reveal how ICL works (Xie et al., 2021; Olsson et al., 2022; Elhage et al., 2021), or investigate the pre-training data to explain ICL’s good performance (Chan et al., 2022; Shin et al., 2022).
Multi-Task Fine-tuning for Cross-Task Generalization Fine-tuning PLMs on a large collection of downstream tasks enables generalization to unseen tasks under zero-shot (Wei et al., 2022; Sanh et al., 2022; Ouyang et al., 2022; Chung et al., 2022) and few-shot (Min et al., 2022b; Chen et al., 2022b; Mishra et al., 2022; Garg et al., 2022) scenarios. However, the performance of multi-task fine-tuning is largely restricted by the diversity of the annotated training tasks (Gu et al., 2022b), which requires massive human efforts to scale up. In addition, direct training on downstream tasks easily brings undesired bias. In this work, we propose to meta-train the model with the intrinsic tasks automatically collected from the large-scale general corpus, which is easier to scale up and introduces little bias.
Pre-training Data Programming The conventional pre-training paradigm trains the model on plain-text corpora with the language modeling objective (Radford et al., 2018, 2019; Brown et al., 2020). Recently works have found that carefully designed pre-training instances can further boost specific abilities like prompt adaption (Gu et al.,
2022a), reasoning (Razeghi et al., 2022), or sentence representation (Levine et al., 2021). Our work studies constructing pre-training instances to improve the PLM’s ICL ability while still maintaining its generalization on various NLP tasks.

heading: 6 Conclusion
text: This paper presents PICL, a framework that exploits the in-context learning ability of PLMs by pre-training models on concatenations of text paragraphs sharing the same “intrinsic tasks” gathered from the large-scale general corpus. In PICL, models learn to perform various intrinsic tasks conditioning on their context while preserving their generalization due to the little bias of the pre-training data. Extensive experiments show that PICL improves the ICL performance on various datasets against several baselines, enabling a 770 M model to outperform a larger model with about 4x parameters while maintaining good generalization across a wide range of tasks. For future work, we would like to consider adding human instructions to our pre-training framework to enhance more abilities of PLMs like zero-shot instruction following.
Limitations
One limitation of our paper is that the exact distribution of the intrinsic tasks in the original corpus and the constructed data is still unknown. Knowing the distribution can offer a better interpretation of the effectiveness of PICL, even of the strong performance of large language models. Besides, although we can find many constructed instances that share obvious intrinsic tasks (see Appendix F), there still exist some instances where the intrinsic tasks are hard to identify. How to better evaluate the contribution of these instances to the ICL ability or designing better filtering approaches to select more informative data for ICL is worth studying.
Our task-semantics encoder inevitably contains some bias because it is trained on downstream datasets, although we have tried to ensure a large number and diversity of the dataset collection. However, the final language model is pre-trained on the general corpus, and we add the full document loss, which eliminates the bias to some extent.
Regarding computing power, we acknowledge that our framework takes relatively large training resources in the retrieval and pre-training process. Therefore, we did not conduct experiments based on extra-large language models.

heading: Acknowledgements
text: This work was supported by the NSFC projects (Key project with No. 61936010 ). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2020GQG0005.

heading: A Details of the Pre-training Corpus
text: This section presents details of the data processing of the pre-training corpus and its statistics.
Data Processing Our pre-training corpus is a merge of OPENWEBTEXT (Gokaslan et al., 2019), WIKICORPUS (Foundation, 2022), and BOOKCORPUS (Zhu et al., 2015), downloaded from the HuggingFace datasets repository1. We first split each document in the corpus into paragraphs with “\n”. To avoid training with too short paragraphs, we concatenate a paragraph with previous paragraphs if the token number after concatenation is lower than 128. We also exclude paragraphs longer than 500 tokens because they are not likely to fit into an instance with more than 1 paragraph. The filtering process in Section 2.3 drops about 24% instances. The licenses of all corpora allow for scientific research.
Statistics We plot the distribution of the mean paragraph length per instance in Figure 7(a) and the distribution of the paragraph number per instance in Figure 7(b). The average paragraph length is 150.0, and the average paragraph number in an instance is 11.7. We can see that the model sees various demonstration numbers in PICL pre-training.

heading: B Details of the Evaluation Data
text: B.1 Few-shot Text Classification
The details of each text classification dataset and the corresponding prompt in evaluation are listed in Table 6. All datasets are downloaded from the HuggingFace datasets repository1. We simplify the evaluation prompts as much as possible to reduce the effect of prompt engineering. Following previous works (Brown et al., 2020; Sanh et al., 2022), the model is evaluated by the ranking score strategy, where we compare the perplexity of each classification label under the model and choose the label with the lowest perplexity. The licenses of all datasets allow for scientific research.
B.2 Instruction Following
The original test split of the benchmark SUPERNATURALINSTRUCTIONS (Wang et al., 2022) contains 119 tasks. We exclude tasks that appear in the training tasks of the task-semantics encoder or whose input length is too long to fit in the context of our model. Our final evaluation includes 105 tasks. A full list of the tasks is shown in Table 8.
We use the same template to combine few-shot examples with task instructions as Wang et al. (2022). The license of this benchmark is Apache License 2.0.

heading: C Details of the Downstream Training Data
text: The downstream datasets we use to train the tasksemantics encoder are a merge of the training data used in (Sanh et al., 2022) and the HR→LR setting in (Min et al., 2022b). All datasets are downloaded from the HuggingFace datasets repository 1 and all prompts come from the PromptSource library (Bach et al., 2022)2. We exclude datasets from the sentiment classification task, the topic classification task, and the natural language inference task because they are included in our text classification evaluation. We finally get a collection of 37 datasets, as listed in Table 5. The licenses of all datasets allow for scientific research.

heading: D More Experimental Details
text: All model checkpoints we used come from the HuggingFace models repository3. The searching interval of each hyper-parameter is listed in Table 4.
1https://huggingface.co/datasets/ 2https://github.com/bigscience-workshop/
promptsource 3https://huggingface.co/models

heading: E More Results
text: E.1 Results on Larger Base Model We test PICL based on the GPT2-xLarge (Radford et al., 2019) with 1.5B parameters. From the results in Figure 7, we can see that PICL is also applicable to larger models, outperforming the baselines based on the same-sized model on most datasets.
E.2 Instruction Following We present the performance comparison between PICL and MetaICL per evaluation task in Figure 8. PICL outperforms MetaICL on 77 / 105 tasks, indicating that PICL ensures the better generalization of the trained model. The name of each task is also listed in Figure 8. We can see that the top three tasks where MetaICL performs the best are:
• doqa_movies_isanswerable, • glue_entailment_classification, • tweetqa_classification,
which are all “Yes/No” classification tasks. The top three tasks where PICL performs the best are:
• winogrande_question_modification_object, • plausible_result_generation, • winowhy_reason_plausibility_detection ,
which are text generation, text generation, and “Correct/Wrong” classification tasks respectively. The four tasks where PICL and MetaICL have the same scores are:
• bard_analogical_reasoning_containers, • copa_commonsense_cause_effect, • winogrande_answer_generation, • bard_analogical_reasoning_trash_or_treasure,
which belong to commonsense reasoning and word analogy tasks.

heading: F Case Studies
text: In Table 9 and 10, we present several cases of the retrieved paragraphs and the corresponding intrinsic tasks. We can see that there exists a large range of intrinsic tasks in the constructed data and many of them do not appear in the training data of the tasksemantics encoder, which shows the generalization of the encoder.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
The section named "Limitations"
3 A2. Did you discuss any potential risks of your work? The section named "Limitations"
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4
3 B1. Did you cite the creators of artifacts you used? Section 3, Appendix B
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Appendix
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? The document to use our code and pre-trained model is in the supplementary materials.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We reported the number of examples, details of train/dev splits, and paragraph lengths of the pre-training corpus.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3, Section 4, Appendix
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: For future work, we would like to consider adding human instructions to our pre-training framework to enhance more abilities of PLMs like zero-shot instruction following. Limitations
One limitation of our paper is that the exact distribution of the intrinsic tasks in the original corpus and the constructed data is still unknown. Knowing the distribution can offer a better interpretation of the effectiveness of PICL, even of the strong performance of large language models. Besides, although we can find many constructed instances that share obvious intrinsic tasks (see Appendix F), there still exist some instances where the intrinsic tasks are hard to identify. How to better evaluate the contribution of these instances to the ICL ability or designing better filtering approaches to select more informative data for ICL is worth studying. Our task-semantics encoder inevitably contains some bias because it is trained on downstream datasets, although we have tried to ensure a large number and diversity of the dataset collection. However, the final language model is pre-trained on the general corpus, and we add the full document loss, which eliminates the bias to some extent. Regarding computing power, we acknowledge that our framework takes relatively large training resources in the retrieval and pre-training process. Therefore, we did not conduct experiments based on extra-large language models