Venue: NAACL 2022

# On the Effect of Pretraining Corpora on

In-context Learning by a Large-scale Language Model

Seongjin Shin\({}^{*,1}\) Sang-Woo Lee\({}^{*,1,2}\) Hwijeen Ahn\({}^{1}\) Sungdong Kim\({}^{2}\)

HyoungSeek Kim\({}^{1}\) Boseop Kim\({}^{1}\) Kyunghyun Cho\({}^{3}\) Gichang Lee\({}^{1}\)

Woomyoung Park\({}^{1}\) Jung-Woo Ha\({}^{1,2}\) Nako Sung\({}^{1}\)

Naver Clova\({}^{1}\) Navera Al Lab\({}^{2}\) NYU\({}^{3}\)

###### Abstract

Many recent studies on large-scale language models have reported successful in-context zero- and few-shot learning ability. However, the in-depth analysis of when in-context learning occurs is still lacking. For example, it is unknown how in-context learning performance changes as the training corpus varies. Here, we investigate the effects of the source and size of the pretraining corpus on in-context learning in HyperCLOVA, a Korean-centric GPT-3 model. From our in-depth investigation, we introduce the following observations: (1) interest learning performance heavily depends on the corpus domain source, and the size of the pretraining corpus does not necessarily determine the emergence of in-context learning, (2) in-context learning ability can emerge when a language model is trained on a combination of multiple corpora, even when each corpus does not result in in-context learning on its own, (3) pretraining with a corpus related to a downstream task does not always guarantee the competitive in-context learning performance of the downstream task, especially in the few-shot setting, and (4) the relationship between language modeling (measured in perplexity) and in-context learning does not always correlate: e.g., low perplexity does not always imply high in-context few-shot learning performance.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

NLP community has been surprised by emergence of in-context learning ability of a large-scale language model (LM) such as GPT-3 Brown et al. (2020) despite no duplication between downstream task data and the pretraining corpus. Indeed, in-context learning uses a natural language description and a few examples to prime a language model. Then the language model can predict the answer of a new example without updating the language model's parameters. Since the release of GPT-3, various large-scale in-context language models have been proposed Black et al. (2021); Kim et al. (2021); Zeng et al. (2021); Rae et al. (2021); Hoffmann et al. (2022); Chowdhery et al. (2022).

There still remain many questions on language models' in-context learning capability despite these successful reports. For example, the relationship between the choice of a pretraining corpus and downstream in-context learning task accuracy is unknown. Previous studies argue pretraining with the corpus similar to the downstream task improves the downstream performance, but these observations are often limited to the case where a pretrained language model is finetuned for the downstream task Gururangan et al. (2020); Lee et al. (2020); Micheli et al. (2020).

In addition, analysis on the relation between the validation complexity of a language model and context learning performance is still less investigated. Previous research on in-context learning implicitly assumes that perplexity is predictive of in-context learning performance by showing scaling law property of their model Kaplan et al. (2020); Brown et al. (2020); Kim et al. (2021). Rae et al. (2021) also use perplexity for the hyperparameter selection on corpus reweighting in the pretraining of their in-context learner. However, their explicit correlations are less discovered.

Motivated by this lack of in-depth analysis on the relationship between in-context learning and corpus properties, we vary the sources and sizes of pretraining corpora and analyze their impact on in-context learning, using HyperCLOVA, which is a Korean-centric large LM. Kim et al. (2021). We mainly discover in-context few-shot learning as in the previous work Kim et al. (2021) but also explore in-context zero-shot learning. We use HyperCLOVA corpus, which is a large-scale pretraining corpus mainly in Korean collected by Kim et al. (2021), as a base corpus from which we derive pretraining corpora for our experiments.



Our major findings include:

* **Corpus Source:** In-context learning performance depends heavily on corpus sources, and with some sources, in-context learning does not work effectively. For example, the model trained only on a subcorpus of blog (Blog) achieves competitive in-context few-shot learning performance, but training on a subcorpus of community website (Cafe) or online news articles (News) hardly yields indextext few-shot learning ability.
* **Corpus Combination:** In-context learning ability can emerge by fusing two corpora, even when each on its own does not result in in-context learning. For example, while training only on KiN corpus, which consists of QnA websites, or Ency corpus, which consists of Encyclopedia websites, in-context few-shot learning ability was not observed, but training on both corpora makes the emergence of in-context few-shot learning.
* **Domain Relevance:** Pretraining with a corpus related to a downstream task seems to help indextext zero-shot learning performance, but is not indicative of the competitive in-context few-shot learning performance. For example, training on only News corpus makes a relatively good in-context zero-shot learning ability on a news-related downstream task, e.g., news topic classification based on its title, KLUE-YNAT Park et al. (2021), but does not yield in-context few-shot learning ability.
* **Perplexity:** Although perplexity and in-context learning accuracies correlate well when training a single model, perplexity alone does not reflect the difference in in-context learning accuracies across different language models. This is prominent particularly when they were trained using different pretraining corpora. For example, Cafe model, the model trained with Cafe corpus, has the second lowest validation perplexity on various domain sources after Blog model, but fails to emerge in-context few-shot learning.

## 2 Related Work

### In-context Learning

Brown et al. (2020) demonstrate the concept of in-context learning, where a few training examples and/or task descriptions are provided together with a new input for a large-scale LM to produce a target of this input, without requiring any parameter update. A few training examples are used in the in-context few-shot learning setting, whereas no training example is used in the in-context zero-shot setting. A few follow-up studies have tried to improve the in-context learning ability Zhao et al. (2021); Holtzman et al. (2021). On the other hand, another group of papers tries to explain the mechanism of in-context few-shot learning Min et al. (2022); Xie et al. (2022).

### Domain Relevance on Pretraining Corpus

Previous studies argue a better downstream accuracy is observed with a pretraining corpus more similar to the downstream task corpus Gururangan et al. (2020); Lee et al. (2020); Micheli et al. (2020). However, these observations are limited to the case where a pretrained language model is finetuned for the downstream task.

There are a few studies on the effects of different corpus on the relationship between pretraining and in-context learning. A notable example is Codex, where GPT-3 is trained on Github corpus so that the model can generate code from comments Chen et al. (2021). However, the corpus used for Codex is limited to code comments and the corresponding code. We study the effect of pretraining corpus on in-context learning performance using various domains.

### Quantity and Quality of Pretraining Corpus

There have been several studies on the quantity and quality of pretraining data. Raffel et al. (2020) conduct an ablation study on different pretraining corpus on T5, and their filtered C4 corpus makes T5 perform better in downstream tasks. As with GPT-3, researchers generally improve the quality of their language model through data filtering Brown et al. (2020); Kim et al. (2021). Our research differs from the existing work in that we focus on in-depth analysis of how the amount of data and the corpus source affect in-context learning.

### Multi-task Learning

Multi-task learning approaches, which explicitly finetune on the in-context learning objective by using numerous NLP tasks, are proposed recently to tackle zero/few-shot transfer to the unseen task 

at test time (Wei et al., 2021; Sanh et al., 2021; Chen et al., 2021b; Min et al., 2021).

Unlike the studies in a finetuning paradigm, many properties of the in-context learning related to pretraining corpus are still unknown. As the previous multi-task studies show that diverse tasks improve the ability of in-context learning, our study shows that diverse pretraining corpora strengthen the ability of in-context learning.

## 3 Task Definition

### Model

We use the variants of HyperCLOVA with various parameter sizes and pretraining corpus. We mainly experiment with models with 1.3B parameters, but we also include the result for 6.9B-sized models. All models have a maximum sequence length of 2,048.

We emphasize that all models use the same vocabulary across all our experiments. We use the morpheme-aware byte-level BPE tokenizer trained with HyperCLOVA corpus Kim et al. (2021) for all models. We train multiple models with different portions of HyperCLOVA corpus to investigate the effects of the source and size of the corpus on in-context learning ability.

### Pretraining with Different Corpus

We analyze the effect of seven subcorpora in the HyperCLOVA corpus: Blog, Cafe, News, Comments, KiN, Modu, and Ency. Table 1 summarizes the characteristics of the subcorpora. Blog, Cafe, and News are taken from blog, community sites, and online news articles of NAVER1, a Korean web portal service, respectively. Comments is the comment threads related to the three subcorpora mentioned above. KiN comes from NAVER's online community QnA service similar to Quora. Ency is a collection of encyclopedic texts including Korean Wikipedia. Modu consists of five public datasets constructed by National Institute of the Korean Language2, including 3.2B of news, 2.1B of written language, 0.4B of spoken language, 0.2B of web corpus, and 0.02B tokens of messenger. Others was excluded to investigate the explicit effects of domain corpus sources on in-context learning because Others is the corpus where various subcorpora are taken from multiple heterogeneous sources. Tables 12 and 13 in Appendix show the examples of seven pretraining corpus in Korean and English, respectively. ALL denotes the original HyperCLOVA corpus including Others.

Footnote 1: [https://www.naver.com/](https://www.naver.com/)

For corpora with less than 150B tokens, we assign 99% of each corpus to the pretraining corpus and randomly extract 10,000 examples from the remaining 1% to the validation corpus for measuring validation perplexity. For corpora with more than 150B tokens, we make the training corpora 150B tokens via random sampling and construct a validation set with 10,000 examples randomly sampled from the remaining. As a result, the maximum training set size of each corpus is 150B tokens.

The validation set for each corpus consists of 10,000 examples and is used for the early stopping of models trained with each corpus. However, we combine all validation set to make the entire validation set contains 70,000 examples for seven domains, and the entire validation set is used for calculating perplexity, as described in Section 3.5.

### Downstream Tasks

We evaluate in-context learning performance of each corpus-specific model on four Korean downstream task datasets used in Kim et al. (2021): NSMC3, KorQuAD Lim et al. (2019), AI Hub translation4, and YNAT Park et al. (2021). NSMC is a binary sentiment classification dataset on movie review. KorQuAD is a machine reading comprehension dataset similar to SQuAD 1.0 Rajpurkar et al. (2016). AI Hub translation dataset consists of Korean-English parallel sentences from news, government websites, legal documents, etc. YNAT is a topic classification problem with seven classes.

Footnote 2: [https://corpus.korean.go.kr/](https://corpus.korean.go.kr/)

Footnote 3: [https://github.com/e9/nsmc](https://github.com/e9/nsmc)

Footnote 4: [https://aihbub.or.kr/aidata/87](https://aihbub.or.kr/aidata/87)

We think that three datasets for downstream tasks are closely rela

\begin{table}
\begin{tabular}{l l c} \hline \hline Name & Description & Tokens \\ \hline Blog & Blog corpus & 273.6B \\ Caffe & Online community corpus & 83.3B \\ News & News corpus & 73.8B \\ Comments & Crawled comments corpus & 41.1B \\ KiiN & Korean OnA website corpus & 27.3B \\ Modu & Collection of five datasets & 6.0B \\ Ency & Encyclopedia corpus & 1.7B \\ Others & Other corpus & 55.0B \\ \hline Total & & 561.8B \\ \hline \hline \end{tabular}
\end{table}
Table 1: Descriptions of HyperCLOVA corpus Kim et al. (2021).

 

pus. Passages which construct KorQuAD are taken from Korean Wikipedia, which is also a part of the Ency. YNAT is a topic classification task of news headlines, so the downstream task is deeply related to the News corpus. A significant portion of parallel sentences for AI Hub translation dataset also comes from news articles. KiN corpus is also related to the translation task. About 2.5% of QnA data in KiN includes Korean questions on the English language, as a foreign language. These question-style passages often include Korean-English sentence pairs in the passage. Vocabulary overlap between downstream tasks and HyperCLOVA corpus is depicted in Figure 1.

### Experimental Details

We try our best to make the same hyperparameter of Kim et al. (2021), including global batch size, training step, maximum sequence length, learning rate, and so on. In our experiments, the models are trained for 72K steps with a global batch size of 1,024. We note that under this setting, the number of tokens that were actually used in pretraining is 150B. Therefore, we set the maximum size of training corpus to 150B as in Section 3.2.

In most experiments, validation perplexity decreases monotonically as training goes on. Thus, we use the checkpoint at 72K step. The only exception is the Ency model. The Ency model has a minimum validation loss at 12K steps, which is likely to be caused by overfitting to pretraining data due to a small size of the data. Therefore, we use early-stopping checkpoints at the 12K steps for the report.

For optimization, AdamW (Loshchilov and Hutter, 2019) with the learning rate of 2.0e-4 and the cosine learning rate scheduling are used. We use the mixed precision training. Models are trained on the Nvidia Superpod which consists of 1,024 A100 GPUs spread across 128 nodes. Using Superpod, it spends around 18 hours to train 1.3B model with 72K steps.

For classification tasks such as NSMC and YNAT, we use a rank classification approach (Wei et al., 2021), where we compare pre-defined outputs ("positive" and "negative") and take the one with higher probability. KorQuAD and AI Hub are free-form completion tasks, where we directly generate output tokens using the greedy decoding.

In the few-shot experiments, the number of shots is set to 70, 4, 4, and 70 for NSMC, KorQuAD, AI Hub, and YNAT, respectively. Downstream tasks are performed 12, 1, 3, and 6 times with different random seeds for NSMC, KorQuAD, AI Hub, and YNAT, respectively. We report the average performance. Random seed influences the sampling of shots from training data and their order. The reason KorQuAD has only one random seed is described in Appendix D. Appendix D also includes the examples of the few-shot prompts used in our experiments. These all experimental settings in the few-shot experiments, from the number of shots to the number of random trials, basically come from the experimental setting of Kim et al. (2021). However, we change the number of trials of YNAT from 3 to 6, because we found that the standard derivation of YNAT is relatively high.

### Measuring Validation Perplexity

We report validation perplexity in various tables and figures to verify our argument. We use the term "PPL" to denote validation perplexities on the validation set. The validation set consists of 70,000 examples from seven corpus sources, as described in Section 3.2. We emphasize that, for calculating PPL, all experiments use **the same vocabulary and validation set**.

In Tables 2 and 4, we use _Italic_ font for the results from a multi-domain model, which is pretrained with two or more mixed corpora. Since a multi-domain model trains more domains than a single-domain model, the PPLs of multi-domain models are generally lower than those of single-domain models. To keep readers from directly comparing PPLs between a single-domain and a multi

Figure 1: Vocabulary overlap ratio between pretraining corpus and downstream task. Top 1,000 nouns are used to calculate the ratio. Nouns are extracted using our in-house part-of-speech tagger.

 

domain model, we use italic font for the results of a multi-domain model.

## 4 Experimental Results

We perform intensive experiments to answer these four main questions:

1. How large do the source and the size of pretraining corpora have the effects on emerging in-context learning ability? (Section 4.2 and 4.3)
2. What is the effect of combining various corpora? (Section 4.4)
3. How large does domain relevance of corpus influence on model performances of the downstream task? (Section 4.5)
4. How strong is the correlation between validation perplexity and in-context learning of language models? (Section 4.6)

### Main Results

Tables 2 and 4 show the in-context few-shot results on various pretraining corpus sources and different corpus combination, respectively. Tables 3 and 5 depict the in-context zero-shot results of some models in Tables 2 and 4, respectively. All results in Tables 2, 3, 4, and 5 come from models with 1.3B parameters. Tables 8 and 9 in Appendix A present the standard derivation values on the results of Tables 2 and 4.

In Tables 2, 3, 4, and 5, Purple-underline denotes the score is below the mean performance value of ALL and Majority baseline in Table 2, and Teablodl denotes the score is above. We use this mean value of Majority and ALL in Table 2 as the performance basis to prevent the in-context learning performance of each model from being distorted by the high basis performance of two classification tasks such as NSMC and YNAT.

Tables 2 and 6 include in-context few-shot results on various pretraining corpus sizes. In Table 6, for example, 56B and 6B correspond to the 1/10 and 1/100 of the original HyperCLOVA corpus with 560B tokens, respectively. The 56B tokens and 6B tokens models are trained with around 3 and 25 epochs, respectively, so that both models can be trained with 72K training steps. On the other hand, Table 2 compares 27B tokens models trained with different corpus sources to show the results in controlled corpus size.

### Effect of Corpus Source

It is noticeable that in-context learning ability emerges differently depending on pretraining corpus sources, as shown in Tables 2, 3, 4, and 5. For example, Blog model makes competitive in-context few-shot learning performance to ALL model, while each of Cafe and News models hardly shows in-context few-shot learning ability from Table 2. It is also noticeable that Modu model performs better than Cafe and News model although the size of Modu corpus is less than 1/10 of Cafe or News corpus, showing the corpus size is not the only factor to predict in-context learning performance. Likewise, it is also interesting that Cafe+News model also shows poor performance despite the same size to Blog and ALL, as shown in Table 4.

These differences in in-context learning are dramatic compared to the finetuning results we expect in general. For a comparative experiment between in-context learning and finetuning in our setting, we also finetuned the experimented models with LoRA Hu et al. (2021). As Table 11 in Appendix C shows, the performance differences in finetuning are much smaller than in the case of in-context learning.

### Effect of Corpus Size

Table 6 shows that reducing the corpus size from 150B to 56B does not decrease the performance severely despite training with 1/10 of corpus. However, the performance degradation of 6B tokens model is remarkable comparing to ALL model. Nevertheless, it is noticeable that 6B tokens model still performs much better than Cafe+News model, which trains 150B tokens of Cafe and News corpus.

We can also see the similar results for three Blog models of different sizes in Table 2. Blog and Blog 54B achieve similar performance. However, like in ALL 6B, Blog 27B performs quite worse than Blog 54B.

Figure 3 shows the comparison between 1.3B-sized model and 6.9B-sized model. In the 6.9B-sized models, the in-context few-shot performance with 56B tokens does not decrease significantly compared to 150B tokens, as in the 1.3B-sized models





### Effect of Combining Corpora

One of our main goals is to investigate the effects of combining multiple corpora from various sources on in-context learning performance. Table 4 shows that in-context few-shot learning ability can be emerged by combining two corpora, even if each of both corpora cannot provide in-context few-shot learning ability. For example, KiN+Ency model succeeds to make in-context learning ability in most tasks, while each of KiN and Ency fails in most tasks. Likewise, Cafe+KiN model succeeds to make in-context few-shot learning ability, while each of Cafe and KiN fails in most tasks. In-context zero-shot abilities of these models follow similar patterns as shown in Table 5.

This phenomenon is related to the argument that in-context learning emerges by multi-task learning. According to the argument, as the language modeling objective function requires a language model to learn variety of next word prediction tasks, the generalization pushes in-context learning ability on unseen tasks. In the example of KiN+Ency model, KiN+Ency may learn in-context learning ability of MRC task, by learning next word prediction tasks of both Ency (Wikipedia) and KiN (QnA).

Unlike these positive cases, we observe that combining corpora does not assure the emergence of competitive in-context learning. For example, from the case of Cafe+News in Table 4, even if the mixed corpus model shows slightly better performance on KorQuAD than each of two corpora, its in-context few-shot performances on NSMC, KorQuAD, and YNAT are still below the basis. Furthermore, the performances on NSMC and YNAT even decrease.

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline Model & Corpus Train & PPL & \multicolumn{2}{c}{NSMC (Acc)} & \multicolumn{1}{c}{KorQuAD} & \multicolumn{3}{c}{AI Hub (BLEU)} & YNAT \\  & & & (Acc) & (EM) & (F1) & Ko\(\rightarrow\)En & En\(\rightarrow\)Ko & (F1) \\ \hline \hline Majority & - & - & 50.35 & 0.0 & 0.0 & 0.0 & 0.0 & 8.26 \\ \hline ALL & 150B & _119.99_ & 84.59 & 56.17 & 73.47 & 6.15 & 23.36 & 59.57 \\ ALL w/o Others & 150B & _119.66_ & 84.59 & 56.49 & 74.20 & 6.14 & 23.21 & 50.76 \\ \hline Blog & 150B & 152.40 & **83.50** & **50.74** & **69.34** & **3.82** & **20.11** & **60.68** \\ Cafe & 82.5B & 170.85 & 57.77 & 3.12 & 14.26 & 2.83 & **16.53** & 11.04 \\ News & 73.1B & 234.78 & 50.72 & 0.14 & 9.96 & 1.10 & **15.88** & 14.36 \\ Comments & 40.7B & 225.39 & **79.78** & 14.69 & 33.33 & 0.79 & 5.06 & **36.17** \\ KiN & 27.0B & 187.80 & 54.73 & 4.85 & 18.99 & **6.81** & **18.16** & 9.23 \\ Modu & 5.9B & 226.01 & **69.91** & **30.20** & **49.29** & 1.21 & 6.13 & **43.27** \\ Ency & 1.7B & 549.40 & 53.81 & 0.71 & 11.88 & 0.58 & 0.69 & 27.99 \\ \hline Blog 54B & 54.0B & 155.69 & **83.06** & **49.13** & **68.10** & **3.93** & **21.12** & **57.97** \\ \hline Blog 27B & 27.0B & 165.60 & **80.27** & 10.91 & 23.41 & **5.35** & **12.32** & **48.19** \\ Cafe 27B & 27.0B & 169.81 & 49.91 & 1.37 & 13.98 & **4.25** & **20.74** & 8.60 \\ News 27B & 27.0B & 239.79 & 50.64 & 0.80 & 8.02 & 2.42 & **15.78** & 27.20 \\ Comments 27B & 27.0B & 229.65 & **80.50** & 13.02 & 31.53 & 1.70 & 3.28 & 25.79 \\ \hline \hline \end{tabular}
\end{table}
Table 2: In-context few-shot learning performance with different pretraining corpus. Models with 1.3B parameters are used. Majority means classifying each label with the primary class, and its score is 0 for KorQuAD and AI Hub. Purple-underline denotes the score is below the mean performance value of ALL and Majority baseline, and Teal-bold denotes the score is above.

\begin{table}
\begin{tabular}{l l r r r r r r r} \hline \hline Model & Corpus Train & PPL & \multicolumn{2}{c}{NSMC (Acc)} & \multicolumn{1}{c}{KorQuAD} & \multicolumn{3}{c}{AI Hub (BLEU)} & YNAT \\  & & & (Acc) & (EM) & (F1) & Ko\(\rightarrow\)En & En\(\rightarrow\)Ko & (F1) \\ \hline \hline Majority & - & - & 50.35 & 0.0 & 0.0 & 0.0 & 0.0 & 8.26 \\ \hline ALL & 150B & _119.99_ & 61.68 & **56.17** & **73.47** & **7.43** & **24.81** & **42.79** \\ \hline Blog & 150B & 152.40 & **75.28** & **50.74** & **69.34** & **5.44** & **22.88** & **49.34** \\ Cafe & 82.5B & 170.85 & **69.38** & 3.12 & 14.26 & **4.34** & **16.44** & **38.12** \\ News & 73.1B & 234.78 & 54.96 & 0.14 & 9.96 & 1.28 & 10.21 & **48.03** \\ Comments & 40.7B & 225.39 & 57.69 & 14.69 & 33.33 & 1.98 & 3.94 & 32.48 \\ KiN & 27.0B & 187.80 & 65.43 & 4.85 & 18.99 & **4.64** & 10.42 & **36.06** \\ Modu & 5.9B & 226.01 & **72.50** & **30.22** & **49.30** & 2.39 & 7.55 & **35.28** \\ Ency & 1.7B & 549.40 & 42.96 & 14.01 & 31.51 & 0.80 & 0.77 & 30.22 \\ \hline \hline \end{tabular}
\end{table}
Table 3: In-context zero-shot performance with different pretraining corpus.





### Effect of Domain Relevance

Speaking of the few-shot results, Table 2 shows that the close relationship between a pretraining corpus and a downstream task does not always guarantee in-context few-shot learning ability on the downstream task. KiN and Ency do not perform well on KorQuAD task, although KorQuAD is an MRC task from Korean Wikipedia, Ency includes Korean Wikipedia, and KiN consists of question answering pair, respectively. Likewise, News does not perform well on YNAT task, although YNAT consists of news headline queries. Table 4 further shows that News+KiN+Ency model shows more degenerated F1 score on YNAT than KiN+Ency, even though a large amount of News corpus is added to News+KiN+Ency model.

For further investigation, we analyze vocabulary statistics of each corpus. Figure 1 shows the vocabulary overlapping ratio between pretraining corpora and downstream tasks. The result shows that high vocabulary overlap between a pretraining corpus and a downstream task does not indicate high downstream task performance. Although the Modu corpus has a large vocabulary overlapping ratio to AI Hub, in-context learning performances of the Modu model on the translation tasks are much lower than Blog and KiN.

The counter example of above supports is AI Hub task performance of KiN model. KiN model learned the pattern of Korean-English sentence pairs, since the corpus includes a lot of Korean questions on English language. While KiN model does not work well in other downstream tasks, the performance on AI Hub translation is competitive and makes the best performance in Ko\(\rightarrow\)En among seven pretraining corpora.

In the zero-shot setting, on the other hand, do

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{\# of tokens} & NSMC & KorQuAD & AlHub (BLEU) & YNAT \\  & (Acc) & (EM) & Ko\(\rightarrow\)En & En\(\rightarrow\)Ko & (F1) \\ \hline \hline
150B & 84.59 & 56.17 & 6.15 & 23.36 & 59.57 \\
56B & 84.35 & 55.13 & 5.47 & 22.98 & 51.89 \\
6B & 74.70 & 36.72 & 3.97 & 17.81 & 30.24 \\ \hline \hline \end{tabular}
\end{table}
Table 6: In-context few-shot learning performance of ALL with different size of the pretraining data. The dataset is randomly sampled from the original corpus.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline Corpus Type & \begin{tabular}{c} Corpus \\ Train \\ \end{tabular} & PPL & \begin{tabular}{c} NSMC \\ (Acc) \\ \end{tabular} & 
\begin{tabular}{c} KorQuAD \\ (FM) \\ \end{tabular} & \multicolumn{2}{c}{AI Hub (BLEU)} & YNAT \\  & & & & Ko\(\rightarrow\)En & En\(\rightarrow\)Ko & (F1) \\ \hline \hline ALL & 150B & _119.99_ & 84.59 & 56.17 & 73.47 & 6.15 & 23.36 & 59.57 \\ \hline \hline \multicolumn{8}{l}{**The Case where In-context few-shot learning Emerges by Combining Two Poor Corpora**} \\ \hline KiN+Ency & 28.7B & _164.69_ & 59.17 & **42.09** & **61.00** & **8.99** & **23.12** & **42.84** \\ Cafe+KiN & 109.5B & _141.92_ & **76.42** & **38.45** & **59.00** & **8.41** & 23.41 & **56.96** \\ \hline \hline \multicolumn{8}{l}{**The Case where In-context few-shot learning Does Not Emerge by Combining Two Poor Corpora**} \\ \hline Cafe+News & 150B & _154.20_ & 54.15 & 8.95 & 22.72 & **4.45** & **17.77** & 8.19 \\ \hline \hline \multicolumn{8}{l}{**The Case of Combining In-context Few-shot Emerging Corpora**} \\ \hline Blog+Comments+Modu & 150B & _144.67_ & **82.82** & **54.94** & **72.27** & **4.09** & **21.17** & **65.01** \\ \hline \hline \multicolumn{8}{l}{**The Case of Adding News into KiN+Ency to Try to Enhance the Performance of YNAT**} \\ \hline News+KiN+Ency & 101.8B & _142.13_ & **75.96** & **35.42** & **55.60** & **8.70** & **23.38** & 27.54 \\ \hline \hline \end{tabular}
\end{table}
Table 4: In-context few-shot learning performance with different corpus combination.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline \multirow{2}{*}{Corpus Type} & \begin{tabular}{c} Corpus \\ Train \\ \end{tabular} & PPL & \begin{tabular}{c} NSMC \\ (Acc) \\ \end{tabular} & 
\begin{tabular}{c} KorQuAD \\ (FM) \\ \end{tabular} & \multicolumn{2}{c}{AI Hub (BLEU)} & YNAT \\  & & & Ko\(\rightarrow\)En & En\(\rightarrow\)Ko & (F1) \\ \hline \hline
150B & _119.99_ & 84.59 & 56.17 & 73.47 & 6.15 & 23.36 & 59.57 \\ \hline \hline \multicolumn{8}{l}{**The Case where In-context few-shot learning Emerges by Combining Two Poor Corpora**} \\ \hline KiN+Ency & 28.7B & _164.69_ & 59.17 & **42.09** & **61.00** & **8.99** & **23.12** & **42.84** \\ Cafe+KiN & 109.5B & _141.92_ & 76.42 & **38.45** & **59.00** & **8.41** & 23.41 & **56.96** \\ \hline \hline \multicolumn{8}{l}{**The Case where In-context few-shot learning Does Not Emerge by Combining Two Poor Corpora**} \\ \hline Cafe+News & 150B & _154.20_ & 54.15 & 8.95 & 22.72 & **4.45** & **17.77** & 8.19 \\ \hline \hline \multicolumn{8}{l}{**The Case of Combining In-context few-shot Emerging Corpora**} \\ \hline Blog+Comments+Modu & 150B & _144.67_ & **82.82** & **54.94** & **72.27** & **4.09** & **21.17** & **65.01** \\ \hline \hline \multicolumn{8}{l}{**The Case of Adding News into KiN+Ency to Try to Enhance the Performance of YNAT**} \\ \hline News+KiN+Ency & 101.8B & _142.13_ & **75.96** & **35.42** & **55.60** & **8.70** & **23.38** & 27.54 \\ \hline \hline \end{tabular}
\end{table}
Table 4: In-context few-shot learning performance with different corpus combination.



main relevance seems to affect more positively. For example, training the News corpus helps in-context zero-shot learning in KLUE-YNAT consistently. As shown in Tables 3 and 5, the models whose training corpus includes the News corpus (i.e., News, Cafe+News, and News+KiN+Ency) even perform better than the model trained whole HyperCLOVA corpus.

In the case of KiN and AI Hub, zero-shot performance increase for AI Hub tasks of the KiN model is less significant than few-shot. However, adding KiN corpus into the pretraining corpus in the experiments of Table 5 (i.e., KiN+Ency, Cafe+KiN, and News+KiN+Ency) makes a consistent performance increase, and the model outperform ALL.

### Perplexity and Downstream Task

Figure 2 presents the scatter plots of PPL (\(x\)-axis) and in-context few-shot learning performance (\(y\)-axis) on five downstream tasks for single corpus models and the ALL model. In Figure 2, we normalized in-context few-shot learning performance by dividing ALL model performance for calibrating various task metrics. Because we observe less explicit tendency of correlation between validation perplexity and in-context performance, we argue that it is difficult to hypothesize better perplexity assures emerging of in-context few-shot learning ability.

According to Table 2, Blog model shows both the lowest PPL and the best in-context learning performance, and Ency model shows both the highest PPL and the worst in-context learning performance. On the contrary, while Cafe model and KiN model shows the second and third lowest PPL, in-context few-shot learning ability was not observed. These results show that the perplexity does not serve as a strong predictor of in-context few-shot learning performance in comparing models trained using different corpora. Table 2 also shows that the corpus size affects in-context few-shot learning performance more than PPL. Blog 27B performs notably worse than Blog, but PPL relatively does not decrease as much.

Speaking of zero-shot results, it seems Table 3 shows that in-context zero-shot learning performances relatively more correlate with perplexity than the few-shot cases. Nevertheless, Modu still has both relatively high perplexity and relatively high in-context zero-shot learning performances.

Table 7 shows validation perplexity scores for

Figure 3: Comparison on two model sizes and two corpus sizes of the original HyperCLOVA corpus such as 1.3B-sized model and 6.9B-sized model, and 56B and 150B tokens. Few-shot results are reported. Green rectangle denotes 1.3B-sized model and blue rectangle denotes 6.9B-sized model.

Figure 2: In-context few-shot learning performance of various corpus models and their PPL. A score of the model is divided by that of ALL to calculate the normalized performance. Blue and red lines denote the performance of ALL model and majority baseline, and the purple line represents the average of both defined in the caption of Table 4.

 

each subcorpus. Each row corresponds to the model and each column corresponds to the validation set's subcorpus. Each validation set except All in Table 7 consists of 10,000 instances, and is the part of our main validation sets, consists of 70,000 instances.

On the other hand, Figure 4 shows that PPL and in-context learning performance correlate well in the perspective of training a single model. We can find that the correlation trends between the cases in the training and the cases between the corpus domain are different.

## 5 Discussion

Our knowledge can be used to increase the performance of in-context learning when the corpus is small or/and there exists demand for collecting more corpus. In the case of XGLM Lin et al. (2021), which is a concurrent work on multilingual GPT-3, achieved better in-context learning performance for many languages. However, it does not reach the performance of a single language model. We hope our observation can give insight into what types of pretraining to be collected more, both for multilingual model and low-resource language model.

Another notable example comes from Gopher Rae et al. (2021), which is a concurrent work on state-of-the-art in-context learner. Rae et al. (2021) determine the ratio between subcorpora based on the perplexity of the validation corpus. They implicitly claim that this ratio results in better downstream task performance, but do not address explicit evidence for this. On the other hand, we are in a position to doubt the strong correlation between perplexity and in-context learning, especially in the few-shot setting. We hope our findings contribute to making better in-context learners along with other research.

## 6 Conclusion

This paper investigates the effects of the source and the size of the training corpus on in-context learning ability, using the HyperCLOVA corpus. Our discoveries include that corpus sources play a crucial role in whether or not in-context learning ability will emerge in a large-scale language model.

One direction for future work is to investigate linguistic properties of corpus sources which make a competitive in-context learning model. For example, quantifying the difference between two corpora can shed light on how to select suitable corpora for NLP practitioners who build large-scale language models. In addition, intensive studies on different corpus sources other than the HyperCLOVA corpus can help understand the properties of in-context learning.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & Blog & Cafe & News & Comments & KiN & Modu & Ency & All \\ \hline Blog & _126.89_ & **201.15** & **83.98** & 599.68 & **138.95** & 98.83 & **108.17** & **152.40** \\ Cafe & **168.03** & _135.37_ & 107.78 & **596.27** & 163.94 & 124.71 & 142.15 & 170.85 \\ News & 281.33 & 432.90 & _60.21_ & 1543.03 & 253.73 & **87.65** & 156.28 & 234.78 \\ Comments & 228.37 & 242.59 & 176.28 & _390.30_ & 164.88 & 196.40 & 239.40 & 225.39 \\ KiN & 232.13 & 278.89 & 150.78 & 689.89 & _50.06_ & 172.78 & 141.45 & 187.80 \\ Modu & 267.59 & 411.35 & 84.36 & 1086.04 & 243.19 & _69.48_ & 136.02 & 226.01 \\ Ensy & 841.53 & 1348.38 & 213.87 & 5889.79 & 543.69 & 266.11 & _73.06_ & 549.40 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Validation perplexity scores per each subcorpus. All denotes the validation perplexity on our main validation set from seven corpus sources. _Italic_ font denotes the validation PPL of their corpus domain, and **Bold** denotes second best after own corpus. Overall, \(\tt{Blog}\) has the best overall validation perplexity in most tasks.

Figure 4: Relation between validation PPL and in-context few-shot learning performance for five downstream tasks as pretraining steps proceed. The results come from the 1.3B-sized ALL model.

 



### Broader Impact Statement

We present multiple pieces of evidence that models using only a part of the pretraining corpus are comparable with those trained with the entire corpus in terms of in-context performances. Although we leave the validation on larger-scale models, such as tens of billion parameters, to future work, our analysis presents a hint to effectively training LMs with smaller corpora. This approach can contribute to alleviating severe energy consumption issues caused by large-scale LMs.

Meanwhile, our study relates to the misuse and fairness of large-scale LMs. For example, reweighting domain-specific corpus might cause LMs to be biased inherent in the domain corpus. Therefore, alleviating domain corpus bias would be a valuable future direction.

## Acknowledgment

The authors thank all the members of CLOVA, AI Lab for devoted supporting and discussion. In particular, they thank Joonsuk Park and Seok Ho Yoon for proofreading.

## References

* S. Black, L. Gao, P. Wang, C. Leahy, and T. Biderman (2021)GPT-Neo: Large Scale Autoregressive Language Modeling with MeshTensorflow. External Links: Link Cited by: SS3.4.
* T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. In NeurIPS, Cited by: SS5.
* M. Chen, R. Zhong, S. Zha, G. Karypis, and H. He (2021)Meta-learning via language model in-context tuning. External Links: Link Cited by: SS6.
* Y. Chen, W. Zhong, S. Zha, G. Karypis, and H. He (2021)Meta-learning via language model in-context tuning. External Links: Link Cited by: SS6.
* A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. Won Chung, C. Sutton, S. Gehrmann, et al. (2022)Palm: scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. Cited by: SS6.
* L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. (2020)The pile: an 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Cited by: SS6.
* S. Gurunargan, A. Marasovic, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith (2020)Don't stop pretraining: adapt language models to domains and tasks. In ACL, Cited by: SS6.
* J. Hoffmann, F. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. Anne Hendricks, J. Welbl, A. Clark, et al. (2022)Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. Cited by: SS6.
* A. Holtzman, P. West, V. Shwartz, Y. Choi, and L. Zettlemoyer (2021)Surface form competition: why the highest probability answer isn t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 7038-7051. External Links: Link Cited by: SS6.
* E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021)Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: SS6.
* J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020)Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: SS6.
* B. Kim, H. Kim, S. Lee, G. Lee, D. Kwak, D. Jeon, S. Park, S. Kim, S. Kim, S. Seo, et al. (2021)What changes can large-scale language models bring? intensive study on hyperclova: billions-scale korean generative pretrained transformers.. In EMNLP, Cited by: SS6.
* J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. Ho So, and J. Kang (2020)BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics36 (4), pp. 1234-1240. Cited by: SS6.
* S. Lim, M. Kim, and J. Lee (2019)Korquad1.0: Korean qa dataset for machine reading comprehension. arXiv preprint arXiv:1909.07005. Cited by: SS6.
* X. Victoria Lin, T. Mihylov, M. Artetxe, T. Wang, S. Chen, D. Simig, M. Ott, N. Gosval, S. Bhosale, J. Du, et al. (2021)Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668. Cited by: SS6.
* I. Loshchilov and F. Hutter (2019)Decoupled weight decay regularization. In ICLR, Cited by: SS6.
* V. Micheli, M. d'Hoffschmidt, and F. Fleuret (2020)On the importance of pre-training data volume for compact language models. In EMNLP,

* Min et al. (2021) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Metaicl: Learning to learn in context.
* Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? _arXiv preprint arXiv:2202.12837_.
* Park et al. (2021) Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyeon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, et al. 2021. Klue: Korean language understanding evaluation. _arXiv preprint arXiv:2105.09680_.
* Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. _arXiv preprint arXiv:2112.11446_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21:1-67.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _EMNLP_.
* Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chahbalani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishaal Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Tibbault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multitask prompted training enables zero-shot task generalization.
* Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners.
* Xie et al. (2022) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning as implicit bayesian inference. In _ICLR_.
* Zeng et al. (2021) Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. 2021. Pangu-\(\alpha\): Large-scale autoregressive pretrained chinese language models with auto-parallel computation. _arXiv preprint arXiv:2104.12369_.
* Zhao et al. (2021) Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. _ArXiv_, abs/2102.09690.

 

Details on Experimental Results

Tables 8 and 9 show standard derivation value on Tables 2 and 4. Table 10 shows score difference with ALL in addition to in-context learning scores on Table 2. Figure 5, supporting Table 7, shows the validation perplexity of different model from different corpus.

## Appendix B Details on Pretraining Corpus

Tables 12 and 13 show example instances of seven pretraining corpus in Korean and English, respectively.

For preprocessing steps of our pretraining corpus, we use HyperCLOVA corpus which is also used in Kim et al. (2021) as described in Section 3.1. Therefore, we share the preprocessing steps of Kim et al. (2021). Appendix A in Kim et al. (2021) describes their preprocessing methods on data descriptoin, data clearing, data anonymization, and data postprocessing.

### Deduplication Preprocess

We additionally introduce the deduplication preprocess of HyperCLOVA corpus, which is used in Kim et al. (2021). The deduplication preprocess was applied to construct HyperCLOVA corpus to prevent explicit duplication within and between subcorpora Kim et al. (2021). According to the response of Kim et al. (2021), they use an in-house search engine and an in-house engineering trick to detect document pairs that are very similar to each other. There are two pipelined steps: (1) removing duplicates within subparts of the corpus, and then (2) removing duplicates between subparts of the corpus. Therefore, documents with high overlap do not exist throughout the documents. Here, the number of subparts is 29. These 29 subparts are categorized into the eight domains we deal with in the paper (i.e., Blog, News, Cafe, Comments, KiN, Modu, Ency, and Others). Overall, there is no explicit overlap between each corpus, since very similar documents have already been removed from the corpus. The overlap between eight HyperCLOVA subcorpora is quite small. There were many overlaps within the subpart of the corpus. However, the overlap between subparts of the corpus was only 0.024% of the total, according to the counts in the second pipelined step of deduplication between subparts.

## Appendix C Experiments on LoRA

Table 11 shows the results of LoRA Hu et al. (2021) finetuning on some models in Tables 2 and 4.

 



## Appendix E Generalization to Other Languages

Someone can ask whether our results can be extended to other languages, including English. We have left experiments on non-Korean language as future work. However, we describe some explanations below to defend our experiments on the Korean language and to discuss why experiments on other languages are practically non-trivial.

First, we think our findings are basically generalizable to other languages. From the perspective of pretraining and in-context learning, fundamental differences between Korean and English were limitedly reported. For example, XGLM (Lin et al., 2021), a concurrent work on, also does not show critical evidence on language-specific properties.

Second, It is non-trivial to control various aspects that are relevant for the QA task. In this section, we present an overview of the results from different perspectives.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Corpus Type & \begin{tabular}{c} Corpus \\ Train \\ \end{tabular} & \begin{tabular}{c} NSMC \\ (Acc) \\ \end{tabular} & \begin{tabular}{c} KorQuAD \\ (EM) \\ \end{tabular} & \begin{tabular}{c} AI Hub (BLEU) \\ (F1) \\ \end{tabular} & \begin{tabular}{c} YNAT \\ Ko\(\rightarrow\)En \\ \end{tabular} & \begin{tabular}{c} EN\(\rightarrow\)Ko \\ \end{tabular} & 
\begin{tabular}{c} (F1) \\ \end{tabular} \\ \hline \hline ALL & 150B & 84.59 & 56.17 & 73.47 & 6.15 & 23.36 & 59.57 \\ \hline \hline \multicolumn{5}{l}{**The Case where In-context learning Emerges by Combining Two Poor Corpora**} \\ \hline KiN+Ency & 28.7B & 59.17(-25.42) & **42.09**(-14.48) & **61.00**(-12.47) & **8.99**(-2.84) & **23.12**(-0.24) & **42.84**(-16.73) \\ Cafe+KiN & 109.5B & **76.42**(-8.17) & **38.45**(-17.72) & **59.00**(-14.47) & **8.41**(-2.26) & **23.41**(-0.06) & **56.96**(-2.61) \\ \hline \hline \multicolumn{5}{l}{**The Case where In-context learning Does Not Emerging by Combining Two Poor Corpora**} \\ \hline Cafe+News & 150B & 54.15(-30.44) & 22.86(-33.31) & 22.72(-50.75) & **4.45**(-1.70) & **17.77**(-5.59) & 8.19(-51.38) \\ \hline \hline \multicolumn{5}{l}{**The Case of Combining In-context Emerging Corpora**} \\ \hline Blog+Comments+Modu & 150B & **82.82**(-1.77) & **54.94**(-1.23) & **72.27**(-1.20) & **4.09**(-2.06) & **21.17**(-2.19) & **65.01**(+5.44) \\ \hline \hline \multicolumn{5}{l}{**The Case of Adding News into KitN+Ency to Try to Enhance the Performance of YNAT**} \\ \hline News+KiN+Ency & 101.8B & **75.96**(-8.63) & **35.42**(-20.75) & **55.60**(-17.87) & **8.70**(-2.55) & **23.38**(-0.02) & 27.54(-32.03) \\ \hline \hline \end{tabular}
\end{table}
Table 10: Table 4 which includes the difference from ALL in parentheses.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \begin{tabular}{c} NSMC \\ (Acc) \\ \end{tabular} & \begin{tabular}{c} AI Hub (BLEU) \\ \end{tabular} & 
\begin{tabular}{c} YNAT \\ (F1) \\ \end{tabular} \\ \hline \hline KiN+Ency & 59.17(-9.34) & **8.99**(-0.31) & **23.12**(-0.40) & **42.84**(-9.01) \\ Cafe+KiN & **76.42**(-4.68) & **8.41**(0.68) & **23.41**(-0.38) & **56.96**(-5.79) \\ \hline Cafe+News & 54.15(-9.30) & **4.45**(-0.12) & **17.77**(-2.19) & 8.19(-5.32) \\ \hline Blog+Comments+Modu & **82.82**(-1.93) & **4.09**(-0.16) & **21.17**(-0.43) & **65.01**(-2.50) \\ \hline News+KiN+Ency & **75.96**(-5.94) & **8.70**(-0.45) & **23.38**(-0.18) & 27.54**(-7.46) \\ \hline \hline \end{tabular}
\end{table}
Table 8: The results of Table 4 with standard deviation in parentheses.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Model & \begin{tabular}{c} NSMC \\ (Acc) \\ \end{tabular} & \begin{tabular}{c} AI Hub (BLEU) \\ \end{tabular} & 
\begin{tabular}{c} YNAT \\ Ko\(\rightarrow\)En \\ \end{tabular} \\ \hline \hline ALL & 84.59(-1.25) & 6.15(-0.16) & 23.36(-0.33) & 59.57(-4.30) \\ ALL w/o Others & 84.59(-1.25) & 6.14(-0.21) & 23.21(-0.45) & 50.76(-11.81) \\ \hline Blog & **83.50**(-2.45) & **3.82**(-0.10) & **20.11**(-0.79) & **60.68**(-5.75) \\ Cafe & 57.77(-10.28) & 2.83(-0.19) & **16.53**(-0.63) & 11.04(-4.43) \\ News & 50.72(-0.56) & 1.10(-0.78) & **15.88**(-0.84) & 14.36(-2.38) \\ Comments & **79.78**(-2.38) & 0.79(-0.01) & 5.09(-0.16) & **36.17**(-3.31) \\ Kill & 54.73(-2.46) & **6.81**(-0.88) & **18.16**(-0.71) & 9.23(-1.96) \\ Modu & **69.91**(-1.84) & 1.21(-0.06) & 6.13(-0.39) & **43.27**(-6.72) \\ Ency & 53.81(-2.22) & 0.58(-0.16) & 0.69(-0.49) & 27.99(-2.38) \\ \hline Blog 54B & **83.06**(-2.26) & **3.93**(-0.20) & **21.12**(-0.19) & **57.97**(-5.72) \\ \hline Blog 27B & **80.27**(-2.28) & **5.35**(-1.95) & **12.32**(-5.68) & **48.19**(-6.71) \\ Cafe 27B & 49.91(-0.29) & **4.25**(-0.35) & **20.74**(-1.57) & 8.60(-2.79) \\ News 27B & 50.64(-3.26) & 2.42(-1.41) & **15.78**(-4.21) & 27.20(-5.86) \\ Comments 27B & **80.50**(-1.44) & 1.70(-0.03) & 3.28(-0.16) & 25.79(-0.27) \\ \hline \hline \end{tabular}
\end{table}
Table 8: The results of Table 2 with standard deviation in parentheses.



pects of corpora for our purpose. Most corpus for in-context few-shot learners comes from crawled website which is not easy to distinguish from its original source. For example, 82% of OpenAI GPT-3 Corpus Brown et al. (2020) is a filtered version of Common Crawl. In this regard, we used relatively a well-refined corpus which consist of several subcorpus from a single web service. (Please see also Section B.1 of this letter.) On the other hand, we have interests to extend our work onto Pile dataset Gao et al. (2020), by controlling the subcorpora in the direction our study pursuits, in the future.

\begin{table}
\begin{tabular}{l c c} \hline \hline Model & NSMC (Acc) & YNAT (F1) \\ \hline \hline ALL & 91.83 & 86.47 \\ \hline Comments & 92.02 & 84.07 \\ Blog & 91.93 & 86.21 \\ Cafe & 91.57 & 85.45 \\ News & 90.62 & 86.57 \\ KiN & 90.89 & 84.46 \\ Modu & 90.60 & 86.31 \\ Ency & 86.93 & 82.37 \\ \hline KiN+Ency & 90.92 & 84.00 \\ Caffe+KiN & 90.99 & 87.52 \\ Cafe+News & 91.37 & 86.37 \\ Blog+Comment+Modu & 88.83 & 87.07 \\ News+KiN+Ency & 91.13 & 86.62 \\ \hline \hline \end{tabular}
\end{table}
Table 11: LoRA finetuning performance on different pretraining corpus and its combination.

 



\begin{table}
\begin{tabular}{l l} \hline \hline  & Example \\ \hline Blog & \\ \hline Cafe & \\ \hline News & \\ \hline Cafe & \\ \hline News & \\ \hline Cafe & \\



\begin{table}
\begin{tabular}{l l} \hline  & Example \\ \hline Blog & Blog \\  & Title: Kids who came to shoot \({}^{\thicksim}\) \\  & Body: Kids who came to my mom’s company as models today. After having fun, playing around, and running around, they read a book and calmed down. Staff members are surprised to see the quiet kids. A smile on my mom’s face! Haha. I’m proud of then because they’re good at reading books, and then I took photos and thank them for doing better than I thought. I thought they do mess it up, but I can't believe that they are looking at the camera. The experience of looking at the camera since early childhood must have helped. Anyway. The kids who came to my mom’s company as models. Thank you \({}^{\thicksim}\). I hope it was a good memory. \\ \hline Cafe & Cafe \\  & Title: Toms women’s shoes (It’s cheaper than what I bought directly “) \\  & Text: Flea market (used), selling, price of 1 won, not used for safety transactions, Tom’s Classic, sales form ID Email, blog, other cafe, other site link, city, dong, exact date of purchase of sales product, desired price of 30,000 to 40,000 won, direct auction transaction. Urgent sale and sell it cheaper than what I bought directly. “1. Tom’s Woman University Ash Gray 45,00 2. Tom’s Woman Chocolate Canvas 40,000. Please contact us..” \\ \hline News & News \\  & Title: Jeon Myeonghwan and Lee Byungki won the Poem Literature Award. \\  & Body: The winners of the 2016 Young Poetry Literature Award and Choi Myung-hee Young Novel Literature Award have been decided. While the awards ceremony was held at Jeonbuk National University’s president’s office on the 1st, Jeon Myung-hyan (second year of Chung-Ang University’s Korean Language Language) who wrote ”the past” in the college category and Hwang Joo-yeon (second year of Gyeongsan Girls’ High School) were selected. Yoon Sun-mi (3rd year of Moohenang, Seoul Digital University), who wrote “From Flowers” in the college category of Choi Myung-hee’s Youth Novel Literature Award, and Yoo Jung-eun (2nd year of Anyang Arts High School), who wrote “Night Flight” in the high school category, were honored. The contest, co-hosted by Jeonbuk National University (President Lee Nam-ho) newspaper broadcasters, Honbul Memorial Society, and Choi Myung-hee Literature Museum (CEO Jang Sung-soo), featured 669 works of 17 people in the poetry category and 116 works of 108 people in the novel category this year. Choi Seung-beom, Yang Byung-ho, Yoo Seung-chul, a member of the Yoo In, and Seo Cheol-won, Hwang Bo-yoon, Jang Mari, Kim So-yoon, and Choi Ki-woo, a member of the novel division, were in charge of the screening. Park Joon-ho, chairman of the Literature Award’s steering committee and weekly newspaper broadcaster, commented, “It was high-quality, and I could get a glimpse of the difficult worries and thoughts of young people looking at the times.” \\ \hline Comments & Conversation \\  & Body: You spend a long day." Ed more for lunch. \\  & It’s brunch. We have to wait for dinner." \\  & What kind of exercise do you do after eating like this? Don’t do sit-ups in the evening. \\  & I’ve been skipping it from time to time. I’m going to visit Anderson next Monday and Tuesday.Let’s go together, right? \\  & Next Monday and Tuesday?** I have to go, I’m going...I’ll make time to go there.“ How are you going to move? \\  & On Tuesday... I have to take the subway, so where will you leave? \\  & I’m departing from Seongsu-dong, Seoul. If you come all the way to Seongsu, I’ll take you." \\ \hline KiN & QnA \\  & Question: Doksilagae? It’s a dialect. What does doksilagae mean? It’s used near Gyeongsang-do. \\  & Text: In Gyeongsang-do dialect, it means tremendously.It’s an advertb to emphasize. \\ \hline Modu & News \\  & Text: It snowed in the spring equinox in Busam.Strong winds are blowing and they’re avoiding it. Yohap News Agency reported that 0.7 centimeters of snow fell on the 21st, the spring equinox in Busan, where snow was not easy, and roads in mountainous areas were controlled. According to the Busan Meteorological Administration, the lowest temperature in the morning in Busan was 1 degree in Daecheng-dong, Jung-gu, where the official observation station is located. Sedaennum-dong, Jung-gu, with an altitude of more than 500m above sea level, recorded minus 1.4 degrees Celisius and Yeongdo-gu recordd minus 0.9 degrees Celisius. As strong winds blev, the sensible temperature dropped to minus 2.6 degrees Celisius. As the morning low temperature crossed below zero, the rain that had been falling all night turned into sleet. The automatic snow observation equipment of the Korea Meteorological Administration in Gijang-gun, Busan recorded a snowfall of 0.7cm. \\ \hline Ency & Document \\  & Title: Seorakmyeon. \\  & Body: Seorak-myeon is a myeson of Gapyeong-gun, Gyeonggi-do, Korea. The area is 141.53 km\({}^{2}\), and the population is 8,986 based on resident registration at the end of December 2016. Seorak-myeon was originally part of Yanppyeong-gun in the south of the Bukhangang River, but was incorporated into Gapyeong-gun in 1942. It has nothing to do with Seorakian Mountain in Gangwon-do. \\ \hline \end{tabular}
\end{table}
Table 13: An example document in Table 12, translated into English by a machine translator.





\begin{table}
\begin{tabular}{l l} \hline Context -> & Ah dubbing. It’s really annoying. voice (Negative) \\  & Hm... I saw the poster and gave elementary school student movie lines...Even overacting isn’t light. (negative) \\  & It was so much fun, so I recommend watching it (negative) \\  & It’s about the prison..Honestly, it’s not fun.Adjusting the rating (negative) \\  & It’s a movie where Simon Peg’s humorous acting stood out!Kristen Dunst, who only looked old in Spider-Man, looked so pretty. (positive) \\  & ... \\  & It did not capture the tension of the original work properly. \\ \hline Correct Answer -> & (negative) \\ \hline Incorrect Answer -> & (positive) \\ \hline \end{tabular}
\end{table}
Table 16: Formatted dataset example for KorQuAD: Machine Reading Comprehension (MRC) (few-shot: 4)

\begin{table}
\begin{tabular}{p{284.5pt} p{390.7pt}} \hline Context -> & Ah dubbing. It’s really annoying. voice (Negative) \\  & Hm... I saw the poster and gave elementary school student movie lines...Even overacting isn’t light. (negative) \\  & It was so much fun, so I recommend watching it (negative) \\  & It’s about the prison..Honestly, it’s not fun.Adjusting the rating (negative) \\  & It’s a movie where Simon Peg’s humorous acting stood out!Kristen Dunst, who only looked old in Spider-Man, looked so pretty. (positive) \\  & ... \\  & It did not capture the tension of the original work properly. \\ \hline Correct Answer -> & (negative) \\ \hline Incorrect Answer -> & (positive) \\ \hline \end{tabular}
\end{table}
Table 14: Formatted dataset example for NSMC. (few-shot: 70)

\begin{table}
\begin{tabular}{c c} \hline Context -> & [A] [B]

* [198] T.-J. Lee, J.-P. Wang, and S.-H. Yang, "A new approach to the classification of a class of visual features from a set of images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 2357-2360, Aug 2014.
* [199] A. Krizhevsky, I. Sutskever, and V. Vanhoucke, "Learning feature representations for large scale image recognition using deep neural networks," in _IEEE International Conference on Acoustics and Speech & Signal Processing Systems (ICASSP)_, pp. 1-10, Sept 2014.
* [200] M. L. LeCun, C.-Y. Chen, and R. Girshick, "ImageNet: A database for object detection with convolutional neural networks," in _International conference on computer vision and pattern recognition_, pp. 1-10, Feb 2014.
* [201] D. P. Kingma, J. Ba, Z. Li, and G. Welling, "Adam: A method for stochastic optimization," in _Advances in Neural Information Processing Systems (NIPS'06)_, vol. 1, no. 1, p. 1130-1138, Dec 2006.
* [202] H. Liu, Y. Zhang, X. Ren, and Q. Weinberger, "Deep residual learning for image recognition," in _Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, Jan 2015.

[MISSING_PAGE_POST]

 

* [184] M. C. Bader, J. D. Krizhevsky, and A. Vedaldi, "A fast learning approach to boosting neural networks for large-scale image recognition," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pp. 2359-2360, 2017.
* [185] R. G. Hinton, S. Hariharan, and I. Sutskever, "Deep residual learning with deep convolutional neural networks," in _Advances in Neural Information Processing Systems_, vol. 10, no. 1, pp. 1035-1040, 2017.
* [186] L. Bottou, Y. Bengio, and P. Salakhutdinov, "Dropout: a simple network for stochastic optimization," _arXiv preprint arXiv:1412.6098_, 2014.
* [187] T. Darrell, E. Farhadi, and F. Fergus, "Image classification via deep convolutional nets," in _IEEE International Conference on Computer Vision and Pattern Recognition_, pp. 1155-1166, 2014.

[MISSING_PAGE_POST]