SlowGuess/NAACL_2022_OCR huggingface dataset:
Unlike these positive cases, we observe that combining corpora does not assure the emergence of competitive in-context learning. For example, from the case of Cafe+News in Table 4, even if the mixed corpus model shows slightly better performance on KorQuAD than each of two corpora, its in-context few-shot performances on NSMC, KorQuAD, and YNAT are still below the basis. Furthermore, the performances on NSMC and YNAT even decrease.
Corpus Type	Corpus Train	PPL	NSMC (Acc)	KorQuAD (EM)	(F1)	AI Hub (BLEU)
Ko→En	En→Ko	YNAT
(F1)
ALL	150B	119.99	84.59	56.17	73.47	6.15	23.36	59.57
The Case where In-context few-shot learning Emerges by Combining Two Poor Corpora
KiN+Ency	28.7B	164.69	59.17	42.09	61.00	8.99	23.12	42.84
Cafe+KiN	109.5B	141.92	76.42	38.45	59.00	8.41	23.41	56.96
The Case where In-context few-shot learning Does Not Emerge by Combining Two Poor Corpora
Cafe+News	150B	154.20	54.15	8.95	22.72	4.45	17.77	8.19
The Case of Combining In-context few-shot Emerging Corpora
Blog+Comments+Modu	150B	144.67	82.82	54.94	72.27	4.09	21.17	65.01
The Case of Adding News into KiN+Ency to Try to Enhance the Performance of YNAT
News+KiN+Ency	101.8B	142.13	75.96	35.42	55.60	8.70	23.38	27.54
Table 4: In-context few-shot learning performance with different corpus combination.
Corpus Type	Corpus Train	PPL	NSMC (Acc)	KorQuAD (EM)	(F1)	AI Hub (BLEU)	YNAT (F1)
\( \mathrm{{Ko}} \rightarrow \mathrm{{En}} \)	\( \mathrm{{En}} \rightarrow \mathrm{{Ko}} \)
ALL	150B	119.99	61.88	56.17	73.47	7.43	24.81	42.79
KiN+Ency	28.7B	164.69	56.78	42.09	61.00	11.51	24.93	37.71
Cafe+KiN	109.5B	141.92	59.27	38.45	59.00	10.12	24.95	45.44
Cafe+News	150B	154.20	66.92	8.95	22.85	3.49	15.77	47.34
Blog+Comments+Modu	150B	144.67	69.15	54.94	72.27	6.06	22.03	48.25
News+KiN+Ency	101.8B	142.13	61.49	35.42	55.60	10.18	24.13	51.89
Table 5: In-context zero-shot learning performance with different corpus combination.
# of tokens	NSMC (Acc)	KorQuAD (EM)	AI Hub (BLEU)	YNAT (F1)
Ko→En	En→Ko
150B	84.59	56.17	6.15	23.36	59.57
56B	84.35	55.13	5.47	22.98	51.89
6B	74.70	36.72	3.97	17.81	30.24
Table 6: In-context few-shot learning performance of ALL with different size of the pretraining data. The dataset is randomly sampled from the original corpus.
4.5 Effect of Domain Relevance
Speaking of the few-shot results, Table 2 shows that the close relationship between a pretraining corpus and a downstream task does not always guarantee in-context few-shot learning ability on the downstream task. KiN and Ency do not perform well on KorQuAD task, although KorQuAD is an MRC task from Korean Wikipedia, Ency includes Korean Wikipedia, and KiN consists of question answering pair, respectively. Likewise, News does not perform well on YNAT task, although YNAT consists of news headline queries. Table 4 further shows that News+KiN+Ency model shows more degenerated F1 score on YNAT than KiN+Ency,
even though a large amount of News corpus is added to News+KiN+Ency model.
For further investigation, we analyze vocabulary statistics of each corpus. Figure 1 shows the vocabulary overlapping ratio between pretraining corpora and downstream tasks. The result shows that high vocabulary overlap between a pretraining corpus and a downstream task does not indicate high downstream task performance. Although the Modu corpus has a large vocabulary overlapping ratio to AI Hub, in-context learning performances of the Modu model on the translation tasks are much lower than Blog and KiN.
The counter example of above supports is AI Hub task performance of KiN model. KiN model learned the pattern of Korean-English sentence pairs, since the corpus includes a lot of Korean questions on English language. While KiN model does not work well in other downstream tasks, the performance on AI Hub translation is competitive and makes the best performance in Ko→En among seven pretraining corpora.
In the zero-shot setting, on the other hand, do
Figure 3: Comparison on two model sizes and two corpus sizes of the original HyperCLOVA corpus such as 1.3B-sized model and 6.9B-sized model, and 56B and 150B tokens. Few-shot results are reported. Green rectangle denotes 1.3B-sized model and blue rectangle denotes 6.9B-sized model.
Figure 2: In-context few-shot learning performance of various corpus models and their PPL. A score of the model is divided by that of ALL to calculate the normalized performance. Blue and red lines denote the performance of ALL model and majority baseline, and the purple line represents the average of both defined in the caption of Table 4.
main relevance seems to affect more positively. For example, training the News corpus helps in-context zero-shot learning in KLUE-YNAT consistently. As shown in Tables 3 and 5, the models whose training corpus includes the News corpus (i.e., News, Cafe+News, and News+KiN+Ency) even perform better than the model trained whole HyperCLOVA corpus.
In the case of KiN and AI Hub, zero-shot performance increase for AI Hub tasks of the KiN model is less significant than few-shot. However, adding KiN corpus into the pretraining corpus in the experiments of Table 5 (i.e.,  $\mathrm{KiN + Ency}$ ,  $\mathrm{Cafe + KiN}$ , and  $\mathrm{News + KiN + Ency}$ ) makes a consistent performance increase, and the model outperforms ALL.
4.6 Perplexity and Downstream Task
Figure 2 presents the scatter plots of PPL  $(x$ -axis) and in-context few-shot learning performance  $(y-$
axis) on five downstream tasks for single corpus models and the ALL model. In Figure 2, we normalized in-context few-shot learning performance by dividing ALL model performance for calibrating various task metrics. Because we observe less explicit tendency of correlation between validation perplexity and in-context performance, we argue that it is difficult to hypothesize better perplexity assures emerging of in-context few-shot learning ability.
According to Table 2, Blog model shows both the lowest PPL and the best in-context learning performance, and Ency model shows both the highest PPL and the worst in-context learning performance. On the contrary, while Cafe model and KinN model shows the second and third lowest PPL, in-context few-shot learning ability was not observed. These results show that the perplexity does not serve as a strong predictor of in-context few-shot learning performance in comparing models trained using different corpora. Table 2 also shows that the corpus size affects in-context few-shot learning performance more than PPL. Blog 27B performs notably worse than Blog, but PPL relatively does not decrease as much.
Speaking of zero-shot results, it seems Table 3 shows that in-context zero-shot learning performances relatively more correlate with perplexity than the few-shot cases. Nevertheless, Modu still has both relatively high perplexity and relatively high in-context zero-shot learning performances.
Table 7 shows validation perplexity scores for
Blog	Cafe	News	Comments	KiN	Modu	Ency	All
Blog	126.89	201.15	83.98	599.68	138.95	98.83	108.17	152.40
Cafe	168.03	135.37	107.78	596.27	163.94	124.71	142.15	170.85
News	281.33	432.90	60.21	1543.03	253.73	87.65	156.28	234.78
Comments	228.37	242.59	176.28	390.30	164.88	196.40	239.40	225.39
KiN	232.13	278.89	150.78	689.89	50.06	172.78	141.45	187.80
Modu	267.59	411.35	84.36	1086.04	243.19	69.48	136.02	226.01
Ency	841.53	1348.38	213.87	5889.79	543.69	266.11	73.06	549.40
Table 7: Validation perplexity scores per each subcorpus. All denotes the validation perplexity on our main validation set from seven corpus sources. Italic font denotes the validation PPL of their corpus domain, and Bold denotes second best after own corpus. Overall, Blog has the best overall validation perplexity in most tasks.
Figure 4: Relation between validation PPL and incontext few-shot learning performance for five downstream tasks as pretraining steps proceed. The results come from the 1.3B-sized ALL model.
each subcorpus. Each row corresponds to the model and each column corresponds to the validation set's subcorpus. Each validation set except All in Table 7 consists of 10,000 instances, and is the part of our main validation sets, consists of 70,000 instances.
On the other hand, Figure 4 shows that PPL and in-context learning performance correlate well in the perspective of training a single model. We can find that the correlation trends between the cases in the training and the cases between the corpus domain are different.
Our knowledge can be used to increase the performance of in-context learning when the corpus is small or/and there exists demand for collecting
more corpus. In the case of XGLM (Lin et al., 2021), which is a concurrent work on multilingual GPT-3, achieved better in-context learning performance for many languages. However, it does not reach the performance of a single language model. We hope our observation can give insight into what types of pretraining to be collected more, both for multilingual model and low-resource language model.
Another notable example comes from Gopher (Rae et al., 2021), which is a concurrent work on state-of-the-art in-context learner. Rae et al. (2021) determine the ratio between subcorpora based on the perplexity of the validation corpus. They implicitly claim that this ratio results in better downstream task performance, but do not address explicit evidence for this. On the other hand, we are in a position to doubt the strong correlation between perplexity and in-context learning, especially in the few-shot setting. We hope our findings contribute to making better in-context learners along with other research.
This paper investigates the effects of the source and the size of the training corpus on in-context learning ability, using the HyperCLOVA corpus. Our discoveries include that corpus sources play a crucial role in whether or not in-context learning ability will emerge in a large-scale language model.
One direction for future work is to investigate linguistic properties of corpus sources which make a competitive in-context learning model. For example, quantifying the difference between two corpora can shed light on how to select suitable corpora for NLP practitioners who build large-scale language models. In addition, intensive studies on different corpus sources other than the HyperCLOVA corpus can help understand the properties of in-context learning.
We present multiple pieces of evidence that models using only a part of the pretraining corpus are comparable with those trained with the entire corpus in terms of in-context performances. Although we leave the validation on larger-scale models, such as tens of billion parameters, to future work, our analysis presents a hint to effectively training LMs with smaller corpora. This approach can contribute to alleviating severe energy consumption issues caused by large-scale LMs.
Meanwhile, our study relates to the misuse and fairness of large-scale LMs. For example, reweighting domain-specific corpus might cause LMs to be biased inherent in the domain corpus. Therefore, alleviating domain corpus bias would be a valuable future direction.
The authors thank all the members of CLOVA, AI Lab for devoted supporting and discussion. In particular, they thank Joonsuk Park and Seok Ho Yoon for proofreading.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. 2021a. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2021b. Meta-learning via language model in-context tuning.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The
pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In ACL.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038-7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dong-pil Seo, et al. 2021. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. In EMNLP.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019. Korquad1.0: Korean qa dataset for machine reading comprehension. arXiv preprint arXiv:1909.07005.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
Vincent Micheli, Martin d'Hoffschmidt, and François Fleuret. 2020. On the importance of pre-training data volume for compact language models. In EMNLP.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Metaicl: Learning to learn in context.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle-moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837.
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, et al. 2021. Klue: Korean language understanding evaluation. arXiv preprint arXiv:2105.09680.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multitask prompted training enables zero-shot task generalization.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning as implicit bayesian inference. In ICLR.
Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. 2021. Pangu- $\alpha$ : Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint arXiv:2104.12369.
Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. ArXiv, abs/2102.09690.
Figure 5: Validation perplexity of different 1.3B-size models in log scale. Color indicates the source of validation corpus.
A Details on Experimental Results
Tables 8 and 9 show standard derivation value on Tables 2 and 4. Table 10 shows score difference with ALL in addition to in-context learning scores on Table 2. Figure 5, supporting Table 7, shows the validation perplexity of different model from different corpus.
B Details on Pretraining Corpus
Tables 12 and 13 show example instances of seven pretraining corpus in Korean and English, respectively.
For preprocessing steps of our pretraining corpus, we use HyperCLOVA corpus which is also used in (Kim et al., 2021) as described in Section 3.1. Therefore, we share the preprocessing steps of Kim et al. (2021). Appendix A in (Kim et al., 2021) describes their preprocessing methods on data descriptoin, data clearing, data anonymization, and data postprocessing.
B.1 Dedduplication Preprocess
We additionally introduce the deduplication preprocess of HyperCLOVA corpus, which is used in (Kim et al., 2021). The deduplication preprocess was applied to construct HyperCLOVA corpus to prevent explicit duplication within and between subcorpora (Kim et al., 2021). According to the response of Kim et al. (2021), they use an in-house search engine and an in-house engineering trick to detect document pairs that are very similar to each other. There are two pipelined steps: (1) removing duplicates within subparts of the corpus, and then (2) removing duplicates between subparts of the
corpus. Therefore, documents with high overlap do not exist throughout the documents. Here, the number of subparts is 29. These 29 subparts are categorized into the eight domains we deal with in the paper (i.e., Blog, News, Cafe, Comments, Kin, Modu, Ency, and Others). Overall, there is no explicit overlap between each corpus, since very similar documents have already been removed from the corpus. The overlap between eight HyperCLOVA subcorpora is quite small. There were many overlaps within the subpart of the corpus. However, the overlap between subparts of the corpus was only  $0.024\%$  of the total, according to the counts in the second pipelined step of dedduplication between subparts.
C Experiments on LoRA
Table 11 shows the results of LoRA (Hu et al., 2021) finetuning on some models in Tables 2 and 4.
D Examples of Few-shot Prompt
Tables 14, 16, 18, and 19 show the example few-shot prompt of NSMC, KorQuAD, AI Hub, and YNAT, respectively. Tables 15, 17, and 20 show the translated version for NSMC, KorQuAD, and YNAT, respectively.
On the other hand, the number of random seed is one for KorQuAD. We explain why evaluation on KorQuAD with many random seeds is difficult, from the perspective of prompt design. The way we make randomness on trials is to change few-shot examples in the prompt. However, in the case of Kim et al. (2021) and in our case, there are no alternative examples to put into the prompt. The prompt examples of KorQuAD are one document and a few question-answer pairs, and not a few document-question-answer triples. In other words, in the prompt of KorQuAD, the number of the document is one. Thus, the document is used for both few-shot question-answer pairs and a query question for the inference. In KorQuAD, there are five corresponding question-answer pairs in each document. In the experimental setting of ours and Kim et al. (2021), four question-answers are put into the prompt and one question is used for the test. Therefore, there are no other question-answer pairs to replace the four pairs.
Model	NSMC (Acc)	AI Hub (BLEU)	YNAT (F1)
\( \mathrm{{Ko}}\rightarrow \mathrm{{En}} \)	\( \mathrm{{En}}\rightarrow \mathrm{{Ko}} \)
ALL	84.59(1.25)	6.15(0.16)	23.36(0.33)	59.57(4.30)
ALL w/o Others	84.59(1.25)	6.14(0.21)	23.21(0.45)	50.76(11.81)
Blog	83.50(2.45)	3.82(0.10)	20.11(0.79)	60.68(5.75)
Cafe	57.77(10.28)	2.83(0.19)	16.53(0.63)	11.04(4.43)
News	50.72(0.56)	1.10(0.78)	15.88(0.84)	14.36(2.38)
Comments	79.78(2.38)	0.79(0.01)	5.06(0.16)	36.17(3.31)
KiN	54.73(4.26)	6.81(0.88)	18.16(0.71)	9.23(1.96)
Modu	69.91(8.41)	1.21(0.06)	6.13(0.39)	43.27(6.72)
Ency	53.81(2.22)	0.58(0.16)	0.69(0.49)	27.99(2.38)
Blog 54B	83.06(2.26)	3.93(0.20)	21.12(0.19)	57.97(5.72)
Blog 27B	80.27(2.28)	5.35(1.95)	12.32(5.68)	48.19(6.71)
Cafe 27B	49.91(0.29)	4.25(0.35)	20.74(1.57)	8.60(2.79)
News 27B	50.64(3.26)	2.42(1.41)	15.78(4.21)	27.20(5.86)
Comments 27B	80.50(1.44)	1.70(0.03)	3.28(0.16)	25.79(7.27)
Table 8: The results of Table 2 with standard deviation in parentheses.
Corpus Type	NSMC (Acc)	AI Hub (BLEU)	YNAT (F1)
Ko→En	En→Ko
KiN+Ency	59.17(9.34)	8.99(0.31)	23.12(0.40)	42.84(9.01)
Cafe+KiN	76.42(4.68)	8.41(0.68)	23.41(0.38)	56.96(5.79)
Cafe+News	54.15(3.90)	4.45(0.12)	17.77(2.19)	8.19(5.32)
Blog+Comments+Modu	82.82(1.93)	4.09(0.16)	21.17(0.43)	65.01(2.90)
News+KiN+Ency	75.96(5.94)	8.70(0.45)	23.38(0.18)	27.54(7.46)
Table 9: The results of Table 4 with standard deviation in parentheses.
Corpus Type	Corpus Train	NSMC (Acc)	KorQuAD	AI Hub (BLEU)	YNAT (F1)
(EM)	(F1)	\( \mathrm{{Ko}} \rightarrow \mathrm{{En}} \)	\( \mathrm{{En}} \rightarrow \mathrm{{Ko}} \)
ALL	150B	84.59	56.17	73.47	6.15	23.36	59.57
The Case where In-context learning Emerges by Combining Two Poor Corpora
KiN+Ency	28.7B	59.17(-25.42)	42.09(-14.08)	61.00(-12.47)	8.99(+2.84)	23.12(-0.24)	42.84(-16.73)
Cafe+KiN	109.5B	76.42(-8.17)	38.45(-17.72)	59.00(-14.47)	8.41(+2.26)	23.41(+0.06)	56.96(-2.61)
The Case where In-context learning Does Not Emerge by Combining Two Poor Corpora
Cafe+News	150B	54.15(-30.44)	22.86(-33.31)	22.72(-50.75)	4.45(-1.70)	17.77(-5.59)	8.19(-51.38)
The Case of Combining In-context Emerging Corpora
Blog+Comments+Modu	150B	82.82(-1.77)	54.94(-1.23)	72.27(-1.20)	4.09(-2.06)	21.17(-2.19)	65.01(+5.44)
The Case of Adding News into KiN+Ency to Try to Enhance the Performance of YNAT
News+KiN+Ency	101.8B	75.96(-8.63)	35.42(-20.75)	55.60(-17.87)	8.70(+2.55)	23.38(+0.02)	27.54(-32.03)
Table 10: Table 4 which includes the difference from ALL in parentheses.
E Generalization to Other Languages
Someone can ask whether our results can be extended to other languages, including English. We have left experiments on non-Korean language as future work. However, we describe some explanations below to defend our experiments on the Korean language and to discuss why experiments on other languages are practically non-trivial.
First, we think our findings are basically generalizable to other languages. From the perspective of pretraining and in-context learning, fundamental differences between Korean and English were limitedly reported. For example, XGLM (Lin et al., 2021), a concurrent work on, also does not show critical evidence on language-specific properties.
Second, It is non-trivial to control various as