SlowGuess/NAACL_2022_OCR huggingface dataset:
Seongjin Shin\*,1 Sang-Woo Lee\*,1,2 Hwijeen Ahn $^{1}$  Sungdong Kim $^{2}$   
HyoungSeok Kim $^{1}$  Boseop Kim $^{1}$  Kyunghyun Cho $^{3}$  Gichang Lee $^{1}$   
Woomyoung Park $^{1}$  Jung-Woo Ha $^{1,2}$  Nako Sung $^{1}$
NAVER CLOVA $^{1}$  Naver AI Lab $^{2}$  NYU $^{3}$
Many recent studies on large-scale language models have reported successful in-context zero- and few-shot learning ability. However, the in-depth analysis of when in-context learning occurs is still lacking. For example, it is unknown how in-context learning performance changes as the training corpus varies. Here, we investigate the effects of the source and size of the pretraining corpus on in-context learning in HyperCLOVA, a Korean-centric GPT-3 model. From our in-depth investigation, we introduce the following observations: (1) in-context learning performance heavily depends on the corpus domain source, and the size of the pretraining corpus does not necessarily determine the emergence of in-context learning, (2) in-context learning ability can emerge when a language model is trained on a combination of multiple corpora, even when each corpus does not result in in-context learning on its own, (3) pretraining with a corpus related to a downstream task does not always guarantee the competitive in-context learning performance of the downstream task, especially in the few-shot setting, and (4) the relationship between language modeling (measured in perplexity) and in-context learning does not always correlate: e.g., low perplexity does not always imply high in-context few-shot learning performance.
NLP community has been surprised by emergence of in-context learning ability of a large-scale language model (LM) such as GPT-3 (Brown et al., 2020) despite no duplication between downstream task data and the pretraining corpus. Indeed, in-context learning uses a natural language description and a few examples to prime a language model. Then the language model can predict the answer of a new example without updating the language model's parameters. Since the release of GPT-3,
various large-scale in-context language models have been proposed (Black et al., 2021; Kim et al., 2021; Zeng et al., 2021; Rae et al., 2021; Hoffmann et al., 2022; Chowdhery et al., 2022).
There still remain many questions on language models' in-context learning capability despite these successful reports. For example, the relationship between the choice of a pretraining corpus and downstream in-context learning task accuracy is unknown. Previous studies argue pretraining with the corpus similar to the downstream task improves the downstream performance, but these observations are often limited to the case where a pretrained language model is finetuned for the downstream task (Gururangan et al., 2020; Lee et al., 2020; Micheli et al., 2020).
In addition, analysis on the relation between the validation perplexity of a language model and in-context learning performance is still less investigated. Previous research on in-context learning implicitly assumes that perplexity is predictive of in-context learning performance by showing scaling law property of their model (Kaplan et al., 2020; Brown et al., 2020; Kim et al., 2021). Rae et al. (2021) also use perplexity for the hyperparameter selection on corpus reweighting in the pretraining of their in-context learner. However, their explicit correlations are less discovered.
Motivated by this lack of in-depth analysis on the relationship between in-context learning and corpus properties, we vary the sources and sizes of pretraining corpora and analyze their impact on in-context learning, using HyperCLOVA, which is a Korean-centric large LM. (Kim et al., 2021). We mainly discover in-context few-shot learning as in the previous work (Kim et al., 2021) but also explore in-context zero-shot learning. We use HyperCLOVA corpus, which is a large-scale pretraining corpus mainly in Korean collected by Kim et al. (2021), as a base corpus from which we derive pretraining corpora for our experiments.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5168 - 5186 July 10-15, 2022 ©2022 Association for Computational Linguistics
Our major findings include:
- Corpus Source: In-context learning performance depends heavily on corpus sources, and with some sources, in-context learning does not work effectively. For example, the model trained only on a subcorpus of blog (Blog) achieves competitive in-context few-shot learning performance, but training on a subcorpus of community website (Cafe) or online news articles (News) hardly yields in-context few-shot learning ability.
- Corpus Combination: In-context learning ability can emerge by fusing two corpora, even when each on its own does not result in in-context learning. For example, while training only on KiN corpus, which consists of QnA websites, or Ency corpus, which consists of Encyclopedia websites, in-context few-shot learning ability was not observed, but training on both corpora makes the emergence of in-context few-shot learning.
- Domain Relevance: Pretraining with a corpus related to a downstream task seems to help in-context zero-shot learning performance, but is not indicative of the competitive in-context few-shot learning performance. For example, training on only News corpus makes a relatively good in-context zero-shot learning ability on a news-related downstream task, e.g., news topic classification based on its title, KLUE-YNAT (Park et al., 2021), but does not yield in-context few-shot learning ability.
- Perplexity: Although perplexity and in-context learning accuracies correlate well when training a single model, perplexity alone does not reflect the difference in in-context learning accuracies across different language models. This is prominent particularly when they were trained using different pretraining corpora. For example, Cafe model, the model trained with Cafe corpus, has the second lowest validation perplexity on various domain sources after Blog model, but fails to emerge in-context few-shot learning.
Brown et al. (2020) demonstrate the concept of in-context learning, where a few training examples
and/or task descriptions are provided together with a new input for a large-scale LM to produce a target of this input, without requiring any parameter update. A few training examples are used in the in-context few-shot learning setting, whereas no training example is used in the in-context zero-shot setting. A few follow-up studies have tried to improve the in-context learning ability (Zhao et al., 2021; Holtzman et al., 2021). On the other hand, another group of papers tries to explain the mechanism of in-context few-shot learning (Min et al., 2022; Xie et al., 2022).
2.2 Domain Relevance on Pretraining Corpus
Previous studies argue a better downstream accuracy is observed with a pretraining corpus more similar to the downstream task corpus (Gururangan et al., 2020; Lee et al., 2020; Micheli et al., 2020). However, these observations are limited to the case where a pretrained language model is finetuned for the downstream task.
There are a few studies on the effects of different corpus on the relationship between pretraining and in-context learning. A notable example is Codex, where GPT-3 is trained on Github corpus so that the model can generate code from comments (Chen et al., 2021a). However, the corpus used for Codex is limited to code comments and the corresponding code. We study the effect of pretraining corpus on in-context learning performance using various domains.
2.3 Quantity and Quality of Pretraining Corpus
There have been several studies on the quantity and quality of pretraining data. Raffel et al. (2020) conduct an ablation study on different pretraining corpus on T5, and their filtered C4 corpus makes T5 perform better in downstream tasks. As with GPT-3, researchers generally improve the quality of their language model through data filtering (Brown et al., 2020; Kim et al., 2021). Our research differs from the existing work in that we focus on in-depth analysis of how the amount of data and the corpus source affect in-context learning.
2.4 Multi-task Learning
Multi-task learning approaches, which explicitly finetune on the in-context learning objective by using numerous NLP tasks, are proposed recently to tackle zero/few-shot transfer to the unseen task
Name	Description	Tokens
Blog	Blog corpus	273.6B
Cafe	Online community corpus	83.3B
News	News corpus	73.8B
Comments	Crowded comments corpus	41.1B
KiN	Korean QnA website corpus	27.3B
Modu	Collection of five datasets	6.0B
Ency	Encyclopedia corpus	1.7B
Others	Other corpus	55.0B
Total	561.8B
Table 1: Descriptions of HyperCLOVA corpus (Kim et al., 2021).
at test time (Wei et al., 2021; Sanh et al., 2021; Chen et al., 2021b; Min et al., 2021).
Unlike the studies in a finetuning paradigm, many properties of the in-context learning related to pretraining corpus are still unknown. As the previous multi-task studies show that diverse tasks improve the ability of in-context learning, our study shows that diverse pretraining corpora strengthen the ability of in-context learning.
We use the variants of HyperCLOVA with various parameter sizes and pretraining corpus. We mainly experiment with models with 1.3B parameters, but we also include the result for 6.9B-sized models. All models have a maximum sequence length of 2,048.
We emphasize that all models use the same vocabulary across all our experiments. We use the morpheme-aware byte-level BPE tokenizer trained with HyperCLOVA corpus (Kim et al., 2021) for all models. We train multiple models with different portions of HyperCLOVA corpus to investigate the effects of the source and size of the corpus on in-context learning ability.
3.2 Pretraining with Different Corpus
We analyze the effect of seven subcorpora in the HyperCLOVA corpus: Blog, Cafe, News, Comments, KiN, Modu, and Ency. Table 1 summarizes the characteristics of the subcorpora. Blog, Cafe, and News are taken from blog, community sites, and online news articles of Naver<sup>1</sup>, a Korean web portal service, respectively. Comments is the comment threads related to the three subcorpora mentioned above. KiN comes
from Naver's online community QnA service similar to Quora. Ency is a collection of encyclopedic texts including Korean Wikipedia. Modu consists of five public datasets constructed by National Institute of the Korean Language $^2$ , including 3.2B of news, 2.1B of written language, 0.4B of spoken language, 0.2B of web corpus, and 0.02B tokens of messenger. Others was excluded to investigate the explicit effects of domain corpus sources on in-context learning because Others is the corpus where various subcorpora are taken from multiple heterogeneous sources. Tables 12 and 13 in Appendix show the examples of seven pretraining corpus in Korean and English, respectively. ALL denotes the original HyperCLOVA corpus including Others.
For corpora with less than 150B tokens, we assign  $99\%$  of each corpus to the pretraining corpus and randomly extract 10,000 examples from the remaining  $1\%$  to the validation corpus for measuring validation perplexity. For corpora with more than 150B tokens, we make the training corpora 150B tokens via random sampling and construct a validation set with 10,000 examples randomly sampled from the remaining. As a result, the maximum training set size of each corpus is 150B tokens.
The validation set for each corpus consists of 10,000 examples and is used for the early stopping of models trained with each corpus. However, we combine all validation set to make the entire validation set contains 70,000 examples for seven domains, and the entire validation set is used for calculating perplexity, as described in Section 3.5.
3.3 Downstream Tasks
We evaluate in-context learning performance of each corpus-specific model on four Korean downstream task datasets used in Kim et al. (2021):  $\mathrm{NSMC}^3$ , KorQuAD (Lim et al., 2019), AI Hub translation $^4$ , and YNAT (Park et al., 2021). NSMC is a binary sentiment classification dataset on movie review. KorQuAD is a machine reading comprehension dataset similar to SQuAD 1.0 (Rajpurkar et al., 2016). AI Hub translation dataset consists of Korean-English parallel sentences from news, government websites, legal documents, etc. YNAT is a topic classification problem with seven classes.
We think that three datasets for downstream tasks are closely related to the HyperCLOVA cor
$^{2}$ https://corpus.korean.go.kr/
<sup>3</sup>https://github.com/e9t/nsmc
4https://aihub.or.kr/aidata/87
<sup>1</sup>https://www.naver.com/
Figure 1: Vocabulary overlap ratio between pretraining corpus and downstream task. Top 1,000 nouns are used to calculate the ratio. Nouns are extracted using our in-house part-of-speech tagger.
pus. Passages which construct KorQuAD are taken from Korean Wikipedia, which is also a part of the Ency. YNAT is a topic classification task of news headlines, so the downstream task is deeply related to the News corpus. A significant portion of parallel sentences for AI Hub translation dataset also comes from news articles. KiN corpus is also related to the translation task. About  $2.5\%$  of QnA data in KiN includes Korean questions on the English language, as a foreign language. These question-style passages often include Korean-English sentence pairs in the passage. Vocabulary overlap between downstream tasks and HyperCLOVA corpus is depicted in Figure 1.
3.4 Experimental Details
We try our best to make the same hyperparameter of Kim et al. (2021), including global batch size, training step, maximum sequence length, learning rate, and so on. In our experiments, the models are trained for 72K steps with a global batch size of 1,024. We note that under this setting, the number of tokens that were actually used in pretraining is 150B. Therefore, we set the maximum size of training corpus to 150B as in Section 3.2.
In most experiments, validation perplexity decreases monotonically as training goes on. Thus, we use the checkpoint at 72K step. The only exception is the Ency model. The Ency model has a minimum validation loss at 12K steps, which is likely to be caused by overfitting to pretraining data due to a small size of the data. Therefore, we use early-stopping checkpoints at the 12K steps for the report.
For optimization, AdamW (Loshchilov and Hutter, 2019) with the learning rate of  $2.0\mathrm{e - }4$  and the cosine learning rate scheduling are used. We use the mixed precision training. Models are trained on the Nvidia Superpod which consists of 1,024 A100 GPUs spread across 128 nodes. Using Superpod, it spends around 18 hours to train 1.3B model with 72K steps.
For classification tasks such as NSMC and YNAT, we use a rank classification approach (Wei et al., 2021), where we compare pre-defined outputs ("positive" and "negative") and take the one with higher probability. KorQuAD and AI Hub are free-form completion tasks, where we directly generate output tokens using the greedy decoding.
In the few-shot experiments, the number of shots is set to 70, 4, 4, and 70 for NSMC, KorQuAD, AI Hub, and YNAT, respectively. Downstream tasks are performed 12, 1, 3, and 6 times with different random seeds for NSMC, KorQuAD, AI Hub, and YNAT, respectively. We report the average performance. Random seed influences the sampling of shots from training data and their order. The reason KorQuAD has only one random seed is described in Appendix D. Appendix D also includes the examples of the few-shot prompts used in our experiments. These all experimental settings in the few-shot experiments, from the number of shots to the number of random trials, basically come from the experimental setting of Kim et al. (2021). However, we change the number of trials of YNAT from 3 to 6, because we found that the standard derivation of YNAT is relatively high.
3.5 Measuring Validation Perplexity
We report validation perplexity in various tables and figures to verify our argument. We use the term "PPL" to denote validation perplexities on the validation set. The validation set consists of 70,000 examples from seven corpus sources, as described in Section 3.2. We emphasize that, for calculating PPL, all experiments use the same vocabulary and validation set.
In Tables 2 and 4, we use Italic font for the results from a multi-domain model, which is pretrained with two or more mixed corpora. Since a multi-domain model trains more domains than a single-domain model, the PPLs of multi-domain models are generally lower than those of single-domain models. To keep readers from directly comparing PPLs between a single-domain and a multi-
domain model, we use italic font for the results of a multi-domain model.
We perform intensive experiments to answer these four main questions:
1. How large do the source and the size of pretraining corpora have the effects on emerging in-context learning ability? (Section 4.2 and 4.3)
2. What is the effect of combining various corpora? (Section 4.4)
3. How large does domain relevance of corpus influence on model performances of the downstream task? (Section 4.5)
4. How strong is the correlation between validation perplexity and in-context learning of language models? (Section 4.6)
Tables 2 and 4 show the in-context few-shot results on various pretraining corpus sources and different corpus combination, respectively. Tables 3 and 5 depict the in-context zero-shot results of some models in Tables 2 and 4, respectively. All results in Tables 2, 3, 4, and 5 come from models with 1.3B parameters. Tables 8 and 9 in Appendix A present the standard derivation values on the results of Tables 2 and 4.
In Tables 2, 3, 4, and 5, Purple-underline denotes the score is below the mean performance value of ALL and Majority baseline in Table 2, and Teal-bold denotes the score is above. We use this mean value of Majority and ALL in Table 2 as the performance basis to prevent the in-context learning performance of each model from being distorted by the high basis performance of two classification tasks such as NSMC and YNAT.
Tables 2 and 6 include in-context few-shot results on various pretraining corpus sizes. In Table 6, for example, 56B and 6B correspond to the 1/10 and 1/100 of the original HyperCLOVA corpus with 560B tokens, respectively. The 56B tokens and 6B tokens models are trained with around 3 and 25 epochs, respectively, so that both models can be trained with 72K training steps. On the other hand, Table 2 compares 27B tokens models trained with different corpus sources to show the results in controlled corpus size.
4.2 Effect of Corpus Source
It is noticeable that in-context learning ability emerges differently depending on pretraining corpus sources, as shown in Tables 2, 3, 4, and 5. For example, Blog model makes competitive in-context few-shot learning performance to ALL model, while each of Cafe and News models hardly shows in-context few-shot learning ability from Table 2. It is also noticeable that Modu model performs better than Cafe and News model although the size of Modu corpus is less than 1/10 of Cafe or News corpus, showing the corpus size is not the only factor to predict in-context learning performance. Likewise, it is also interesting that Cafe+News model also shows poor performance despite the same size to Blog and ALL, as shown in Table 4.
These differences in in-context learning are dramatic compared to the finetuning results we expect in general. For a comparative experiment between in-context learning and finetuning in our setting, we also finetuned the experimented models with LoRA (Hu et al., 2021). As Table 11 in Appendix C shows, the performance differences in finetuning are much smaller than in the case of in-context learning.
4.3 Effect of Corpus Size
Table 6 shows that reducing the corpus size from 150B to 56B does not decrease the performance severely despite training with 1/10 of corpus. However, the performance degradation of 6B tokens model is remarkable comparing to ALL model. Nevertheless, it is noticeable that 6B tokens model still performs much better than Cafe+News model, which trains 150B tokens of Cafe and News corpus.
We can also see the similar results for three Blog models of different sizes in Table 2. Blog and Blog 54B achieve similar performance. However, like in ALL 6B, Blog 27B performs quite worse than Blog 54B.
Figure 3 shows the comparison between 1.3B-sized model and 6.9B-sized model. In the 6.9B-sized models, the in-context few-shot performance with 56B tokens does not decrease significantly compared to 150B tokens, as in the 1.3B-sized models.
Model	Corpus Train	PPL	NSMC (Acc)	KorQuAD (EM)	(F1)	AI Hub (BLEU)	YNAT (F1)
\( \mathrm{{Ko}} \rightarrow \mathrm{{En}} \)	\( \mathrm{{En}} \rightarrow \mathrm{{Ko}} \)
Majority	-	-	50.35	0.0	0.0	0.0	0.0	8.26
ALL	150B	119.99	84.59	56.17	73.47	6.15	23.36	59.57
ALL w/o Others	150B	119.66	84.59	56.49	74.20	6.14	23.21	50.76
Blog	150B	152.40	83.50	50.74	69.34	3.82	20.11	60.68
Cafe	82.5B	170.85	57.77	3.12	14.26	2.83	16.53	11.04
News	73.1B	234.78	50.72	0.14	9.96	1.10	15.88	14.36
Comments	40.7B	225.39	79.78	14.69	33.33	0.79	5.06	36.17
KiN	27.0B	187.80	54.73	4.85	18.99	6.81	18.16	9.23
Modu	5.9B	226.01	69.91	30.20	49.29	1.21	6.13	43.27
Ency	1.7B	549.40	53.81	0.71	11.88	0.58	0.69	27.99
Blog 54B	54.0B	155.69	83.06	49.13	68.10	3.93	21.12	57.97
Blog 27B	27.0B	165.60	80.27	10.91	23.41	5.35	12.32	48.19
Cafe 27B	27.0B	169.81	49.91	1.37	13.98	4.25	20.74	8.60
News 27B	27.0B	239.79	50.64	0.80	8.02	2.42	15.78	27.20
Comments 27B	27.0B	229.65	80.50	13.02	31.53	1.70	3.28	25.79
Table 2: In-context few-shot learning performance with different pretraining corpus. Models with 1.3B parameters are used. Majority means classifying each label with the primary class, and its score is 0 for KorQuAD and AI Hub. Purple-underline denotes the score is below the mean performance value of ALL and Majority baseline, and Teal-bold denotes the score is above.
Model	Corpus Train	PPL	NSMC (Acc)	KorQuAD	AI Hub (BLEU)	YNAT (F1)
(EM)	(F1)	\( \mathrm{{Ko}} \rightarrow \mathrm{{En}} \)	\( \mathrm{{En}} \rightarrow \mathrm{{Ko}} \)
Majority	-	-	50.35	0.0	0.0	0.0	0.0	8.26
ALL	150B	119.99	61.68	56.17	73.47	7.43	24.81	42.79
Blog	150B	152.40	75.28	50.74	69.34	5.44	22.88	49.34
Cafe	82.5B	170.85	69.38	3.12	14.26	4.34	16.44	38.12
News	73.1B	234.78	54.96	0.14	9.96	1.28	10.21	48.03
Comments	40.7B	225.39	57.69	14.69	33.33	1.98	3.94	32.48
KiN	27.0B	187.80	65.43	4.85	18.99	4.64	10.42	36.06
Modu	5.9B	226.01	72.50	30.22	49.30	2.39	7.55	35.28
Ency	1.7B	549.40	42.96	14.01	31.51	0.80	0.77	30.22
Table 3: In-context zero-shot performance with different pretraining corpus.
4.4 Effect of Combining Corpora
One of our main goals is to investigate the effects of combining multiple corpora from various sources on in-context learning performance. Table 4 shows that in-context few-shot learning ability can be emerged by combining two corpora, even if each of both corpora cannot provide in-context few-shot learning ability. For example,  $\text{KiN+Ency}$  model succeeds to make in-context learning ability in most tasks, while each of  $\text{KiN}$  and  $\text{Ency}$  fails in most tasks. Likewise,  $\text{Cafe+KiN}$  model succeeds to make in-context few-shot learning ability, while each of  $\text{Cafe}$  and  $\text{KiN}$  fails in most tasks. In-context zero-shot abilities of these models follow similar patterns as shown in Table 5.
This phenomenon is related to the argument that in-context learning emerges by multi-task learning.
According to the argument, as the language modeling objective function requires a language model to learn variety of next word prediction tasks, the generalization pushes in-context learning ability on unseen tasks. In the example of KiN+Ency model, KiN+Ency may learn in-context learning ability of MRC task, by learning next word prediction tasks of both Ency (Wikipedia) and KiN (QnA).