HyperT5: Towards Compute-Efficient Korean Language Modeling
Year: 2023

Abstract:
Pretraining and fine-tuning language models have become the standard practice in industrial natural language processing (NLP), but developing and deploying general-purpose language models without the abundant computation or data resources is a real-world issue faced by smaller organizations or communities whose main focus is languages with less accessible resources (e.g., non-English). This paper explores the sequence-to-sequence (seq2seq) language model architecture as a more practical and compute-efficient alternative to the decoder-oriented approach (e.g., GPT-3), accompanied by novel findings in compute-optimality analyses. We successfully trained billion-scale Korean-language seq2seq language models that strongly outperform other competitive models in Korean benchmarks. Moreover, we demonstrate that such language models can be more efficiently utilized by employing a heavy pre-finetuning strategy, by showcasing a case study on dialog-task adaptation. Our case study shows that adopting language models with more readily available domain-specific unlabeled data greatly improves fine-tuning data efficiency in lowresource settings.

heading: 1 Introduction
text: Pretraining large-scale Transformer-based language models and finetuning them for specific tasks have become the cornerstone of modern NLP pipelines. Among various Transformer-based language model architectures proposed in the field, generative decoder-based architectures, such as the GPT family (Brown et al., 2020), have gained more traction from their impressive ability to scale well into large language models (LLMs) (Kaplan et al., 2020; Chowdhery et al., 2022) and follow highlevel natural language instructions with few or even in the absence of demonstrations (Wei et al., 2022).
∗ Equal contributions.
However, acquiring pretrained language models (PLMs) is a data- and compute-intensive process (Patterson et al., 2021), which many organizations cannot afford to pursue. This disparity is exacerbated for the communities of non-English or nonLatin languages (e.g., Korean) that have limited access to resources and share fewer linguistic features with English, making the cross-lingual transfer from the top language more challenging (Scao et al., 2022).
As a more cost-efficient alternative to the pure generative architecture, the sequence-to-sequence (seq2seq) Transformer (T5) (Raffel et al., 2022) may offer a reasonable middle ground between the generative LM and the encoder-oriented architecture (e.g., BERT (Devlin et al., 2019)), which are known to lack robustness in generation abilities. Additionally, T5 has been demonstrated to
412
be good few-shot learners (Liu et al., 2022a) and task/domain adapters (Aribandi et al., 2022; Gupta et al., 2022).
This paper aims to provide an industrial perspective on the language model pretraining strategies with small- to medium-scale budgets. We conduct compute-optimality analyses to find the optimal pretraining strategy given our compute budget and argue that the text-to-text Transformer architecture is a superior approach compared to decoder-only models under restricted compute resources (§5.1). Based on this finding, we share our experience with training HyperT5 (§3), the state-of-the-art seq2seq Transformer for the Korean language (Figure 1). Moreover, we showcase how HyperT5 can be further refined to improve data and modeling efficiency in specific domains (i.e. dialogs) using relatively abundant unlabeled resources (§4).

heading: 2 Related Work
text: PLMs and Efficiency As we gain more understanding of core PLM architectures (Devlin et al., 2019; Brown et al., 2020; Raffel et al., 2022) and their scaling laws (Kaplan et al., 2020), research efforts for improving their training efficiencies have diverged in various directions, including improving the scaling curve (Tay et al., 2022; Chung et al., 2022), maximizing optimality (Hoffmann et al., 2022), and efficient fine-tuning (Lester et al., 2021; Hu et al., 2022). Our work offers a comprehensive overview and case study of optimizing the language modeling efficiency for a less resourceful language.
Non-English Language Models Previously, significant works have explored pretraining language models on multiple languages to support lowresource languages and maximize cross-lingual knowledge transfer without explicit supervision (Devlin et al., 2019; Conneau and Lample, 2019; Xue et al., 2021; Scao et al., 2022). Recently, lan-
guage models that target specific non-English languages started to become more common (Zeng et al., 2021; Kim et al., 2021a; Nagoudi et al., 2022; Fuadi et al., 2023), especially low-resource or nonLatin languages that share fewer commonalities with English. There have been several Korean textto-text Transformers proposed in the past, such as KoBART* and KE-T5 (Kim et al., 2021b), but our work, among other Korean seq2seqs, is the first to systematically achieve powerful billion-scale seq2seq models and conduct extensive analyses in terms of efficiency and performance.
Dialog-Oriented Language Models Adapting language models for the purpose of building dialog agents has been a long-standing goal in the language modeling community (Zhang et al., 2020; Adiwardana et al., 2020; Roller et al., 2021). However, from the industrial application perspective, building dialog response generators is not the only task that can benefit from the advances in language modeling. In a more recent line of work, several approaches have been proposed to prepare language models for various dialog-related tasks (Mehri et al., 2019; Gu et al., 2021; Chen et al., 2022). In parallel to dialog adaptation, multi-task fine-tuning is another line of work that covers dialog-related tasks, as a subset of dialog-related tasks is included in the multi-task set, and expanding the task set to cover dialog-related tasks is trivial (Sanh et al., 2022; Aribandi et al., 2022).

heading: 3 HyperT5
text: This section describes the details of the pretraining corpus, pretraining strategy, and evaluation methods.

heading: 3.1 Pretraining Corpus
text: Inspired by the pretraining corpus proposed by Kim et al. (2021a), we design our pretraining data to cover a wide range of domains and data distributions to ensure that the model trained on top of the corpus will achieve robustness and generalizability. Various sources of written and spoken texts are included in the corpus (Table 1), although online web texts take a large portion of the data (Table 5). While online texts are certainly vulnerable to the compound effect of biases, the collectively massive and unfiltered nature provides a comprehensive impression of the language distribution
*https://github.com/SKT-AI/KoBART
(Scao et al., 2022). We conducted additional studies to investigate the feasibility of data composition re-adjustment through sampling (Appendix A.1). However, we found that the benefit was not clearcut.

heading: 3.2 Pretraining Setup
text: Our research employs the transformer encoderdecoder architecture, similar to T5 of Google (Raffel et al., 2022). However, we have opted for the T5.1.1 structure, a variation of T5, due to its superior performance based on experimentation results. It is worth noting that our HyperT51.7B model size is not a derivative of Google’s T5, but rather an interpolation of the LARGE and the 3B model. Detailed information on the configuration of different model sizes can be found in Table 2.
For all model has been pre-trained on a total of 300B tokens, utilizing the replace corrupted spans method proposed by Google’s T5 as one of their unsupervised objectives. Specifically, we set the corruption rate to 15%, while maintaining a mean span length of 3.
Furthermore, we employed the inverse square root learning rate schedule with 10k warmup steps at a learning rate of 0.01 when using the Adafactor optimizer (Shazeer and Stern, 2018). Both of our pretrained models were trained using a batch size of 1024 and a maximum sequence length of 512 for the encoder and decoder, respectively.
By using distributed data-parallel (Li et al.), we were able to parallelize the training process across multiple GPUs, effectively reducing the overall training time and enabling us to train larger models with higher performance. Specifically, we used 64 A100 GPUs for the small to large models and 1024 A100 GPUs for the 1.7B and 3B models.

heading: 3.3 Evaluation Methods
text: Benchmark The primary objective of the HyperT5 evaluation is to address various natural language processing tasks specific to the Korean language in a reproducible way. To quantify the effec-
tiveness of our model in these tasks, we designed a series of benchmarking experiments that cover a wide range of tasks. The detailed components of our benchmark are described in Appendix A.2. Note that while all of our benchmark datasets are publicly available for reproducibility, some datasets (YNAT, KLUE-NLI, KLUE-STS, KorQuAD) have not made the test set publicly available, hence some of the report values are based on the development or validation set where the test set is inaccessible.
Baselines We compare not only structures that are identical to ours but also encoder and decodeexclusive architectures. Models based on the BERT and RoBERTa architecture released by KLUE (Park et al., 2021) are encoder-only models specialized for natural language understanding. On the other hand, HyperCLOVA (Kim et al., 2021a) is a decoder-only structure like GPT. Note that HyperCLOVA does not provide fine-tuning results, and thus, we compare the ICL and P-tuning (Liu et al., 2022b) results reported for this model. We also compare three models with the same structure as our model. KoBART has the encoder-decoder structure but follows the learning method and details of BART (Lewis et al., 2019). The mT5 (Xue et al., 2021) and KE-T5 (Kim et al., 2021b) models share the exact same structure as our HyperT5 model, with the difference being that mT5 is a multilingual model and KE-T5 is a Korean and English cross-lingual model.

heading: 3.4 Evaluation Results
text: Main Benchmark Results On our Korean benchmark, HyperT5 achieves state-of-the-art performances across all tasks (Table 3), outperforming other seq2seq architectures by a large margin. Specifically, the smallest version of our model (97M) was able to perform on par with the largest KE-T5 (large) on the average benchmark (87.96 vs 88.61). Compared to large-scale decoder architectures, our largest model (3B) is still able to outperform the 39B-scale HyperCLOVA with p-tuning (93.29 vs 93.00). Although a more comprehensive benchmark is desirable, the preliminary results on NSMC suggest that our approach has an advantage in scaling efficiency for the current compute-budget range (§5.1).
Parameter-Efficient Fine-Tuning To understand how our model can be further efficiently finetuned using parameter-efficient fine-tuning (PEFT) techniques, we benchmarked HyperT5 models that

heading: Model Params. NSMC YNAT KLUE-NLI KLUE-STS Avg.
text: are fine-tuned using LoRA (Hu et al., 2022) and Ladder-Side Tuning (LST) (Sung et al., 2022), respectively, and compared the performances against the full fine-tuning results in Table 3. Results show that the performance degradation of employing PEFT compared to the full fine-tuning baseline is less than 5% on average, while the ratio of parameters used for training is less than 2.3%. And as the model scales larger, the issue of performance degradation is relatively alleviated, falling to 0.39% for HyperT53B with LoRA. Model scaling and the specific PEFT technique to employ will be the key strategic factors for large-scale deployment.

heading: 4 Case Study: Efficient Adaptation for Dialog-Oriented Tasks
text: Domain and task-family adaptation can further improve the utilization of PLMs in low-resource settings (Maronikolakis and Schütze, 2021). This section explores the use case for adapting HyperT5 to dialog-related tasks.

heading: 4.1 Training Setup
text: For dialog adaptation, we propose to heavily train HyperT5 on a 1B-token unlabeled dialog-oriented data, with the multiple utterance masking (MUM) objective in the curriculum learning setting. We take the replace corrupted spans method to a more challenging strategy, MUM, to help the model hold a better understanding of dialog structures. During training, multiple utterances are randomly masked per dialog session with a pre-defined corruption rate. We further adopt curriculum learning to gradually raise the training difficulty by increasing the MUM corruption rate. HyperT5 models, from small to large, are trained for 5 epochs with a global batch size of 64 using 2 A100 GPUs. Like pretraining, the dropout rate is set to 0. MUM corruption rate sweeps sigmoidally from 5% to 40%.
Training Data We collect a dialog-oriented training corpus from both open-sourced and proprietary Korean dialog datasets (Appendix B.1). The corpus consists of 3.3M dialog sessions† in various domains (e.g. social chats, customer service, broadcast transcripts, etc.). The resulting corpus provides a wide range of topics and aspects of different dialog-oriented tasks, making it suitable for dialog adaptation.
†We preprocessed the dialog corpus by truncating and splitting the original dialogs into up to 20-turn sessions.

heading: 4.2 Evaluation Methods
text: We conducted an extended series of benchmarking experiments for our dialog-adapted models (DialogHT5) in both scarce and full data settings. The benchmark results consist of three generative tasks, i.e., dialog in-filling (DI), dialog response generation (DR), and dialog summarization (DS), and one discriminative task, i.e., dialog classification (DC). We only use the open-sourced Korean dialog-oriented datasets from AI Hub‡. AI Hub task-oriented dialog (ToD) and open-domain dialog (ODD) datasets are used for both DI and DR, while AI Hub broadcasting media transcript summarization dataset (Script-Summ) for DS. Since the AI Hub ToD dataset is annotated with dialog topics, we used the dataset for DC (Appendix B.2).

heading: 4.3 Dialog Benchmark Results
text: Data Scarce Experiments To investigate the data efficiency of DialogHT5, we first compare their performance against HyperT5 over scarce data settings. We benchmarked DialogHT5 and HyperT5 as sweeping the number of training samples from 0.1k to 5k. Results describe that DialogHT5 mostly achieves a higher score against HyperT5 (more details in Appendix B.3). Especially, DialogHT5 outperforms HyperT5 with the gain of up to 8 R1 score in the dialog in-filling task, as shown in Figure 2. DialogHT5SMALL can save the in-domain data resource approximately ten times to score tie with HyperT5SMALL of 5k training samples (Figure 2a), which highlights the effectiveness of dialog adaptation.
Full Data Experiments Full data benchmark results show that DialogHT5 obtains higher performance compared to HyperT5 in all the experiments (Appendix B.4). Generative tasks show a gain of
‡https://aihub.or.kr/
up to 0.6 R1 score whereas the discriminative task shows a gain of 0.7 F1 score.

heading: 5 Discussions

heading: 5.1 Compute-Optimality Analysis
text: To investigate whether the model configurations we experimented with are optimal given our pretraining compute budget and the pretraining tokens, we conducted compute-optimality analyses, similar to the work done by Hoffmann et al. (2022). The loss curves of our pretraining experiments were smoothed and interpolated as shown in Figure 3a. Using the curves, we map out the optimal model sizes for each given compute level. However, due to the very small number of model-size samples, we need to normalize the optimal model-size data points by selecting the mid-point of each optimality segment§, as shown in Figure 3b. After fitting the regression line (r2 = 0.988), we discover that the size of our largest model lies very close to the predicted optimal model size for our compute budget. Moreover, the predicted optimal model size (dashed line in the same figure) for the decoderonly architecture is significantly smaller (at 448M), but our benchmark results on NSMC (Table 3) show that small-scale decoder LMs (i.e., HyperCLOVA) falls behind in terms of performance, supporting the notion that seq2seq architectures are
§The compute range, where the smallest and the largest model sizes are chosen to be the optimal model, is omitted to prevent skewness.
more economically viable for small and mediumscale compute budgets¶.

heading: 5.2 Practical Advantages of Seq2Seq
text: Apart from the quantitative benefits in performance and efficiency demonstrated throughout the paper, Seq2Seq offers additional practical and real-world benefits. First, the encoder-decoder framework produces a parameter-efficient text encoder as a byproduct, which can be utilized for extracting features and encoding purposes (Ni et al., 2022; Liu et al., 2021). Specifically, the encoder module extracted from seq2seq is capable of producing highquality text embeddings superior to ones produced from encoders of similar sizes (Ni et al., 2022).
Second, the text-to-text architecture reduces the software complexity and management costs for large-scale deployment, as a result of (1) the unified nature of the input and output format, (2) the separation of the input and output sequences inherently supported by the encoder-decoder architecture, and (3) better parameter-efficiency. This translates to fewer engineering resources to support the same level of deployment scalability. The unified text nature of the data format allows existing deployment infrastructures to be easily expanded to handle new tasks. Also, the inherent distinct twopart architecture enables simpler and more streamlined serving infrastructure. Furthermore, due to the steeper model-scaling curve exhibited by decoder architectures, text-to-text transformers incur fewer operating costs to maintain the same quality of services.

heading: 6 Conclusion
text: In this paper, we introduced HyperT5 and DialogHT5 as state-of-the-art on Korean language modeling. We also demonstrated the feasibility of performing resource-aware strategization for language models. Through the compute-optimality analyses, we found that the seq2seq architecture may be more cost-efficient than decoders below a certain compute-budget threshold. For future work, we wish to generalize the domain adaptation approach and study the efficacy of multi-task learning (Aribandi et al., 2022) from the industrial perspective. Furthermore, we look forward to conducting comprehensive investigations into cross-architectural optimality.
¶Conversely, this means that decoder-only architectures scale better with larger compute budgets (Figure 3b).

heading: Ethics Statement
text: The authors are aware that the language models proposed in this paper, either pretrained from our pretraining corpus or heavily fine-tuned using the dialog corpus (Appendix B.1), are all subject to social and unethical biases depending on the way the corpora were prepared. Internally, the authors and the relevant members of the affiliated organization are actively working to make sure that the deployed language models do not generate ethically questionable content that may cause harm or stress to the end user. The specific set of actions that we take include but are not limited to,
• Employing automated models to detect unethical content and perform automatic adversarial attacks on the language model before deploying them into services and products.
• Under safe and strict ethical guidelines, conducting human studies to identify prompts that could potentially cause the language model to generate unethical content. (red-teaming)
• Establishing strategies to mitigate or amend ethical issues exhibited by the language models raised from automated and human surveys.

heading: A Supplementary Materials Regarding HyperT5 Pretraining

heading: A.1 Experiments on Corpus Composition Re-sampling

heading: Model (corpus) NSMC YNAT KLUE-NLI KLUE-STS Abs. Summ. Avg.
text: During pretraining, language models consume billions of weakly preprocessed tokens, which may impact the model performance to varying degrees. For example, web-crawled data which take up a large portion of the pretraining corpus is relatively noisy, thus prioritizing certain data sources that are thought to be dense with information may help to improve training efficiency, i.e., the number of tokens needed to converge towards a reasonable level of performance.
To investigate our hypothesis, we create an experimental pretraining corpus (Table 5) and made sure that the proportions of data sources that provide “hard” knowledge (e.g., books and news) are significantly higher by under-sampling other data types (Rae et al., 2021). Additionally, we augment the experimental pretraining corpus with English data sources, leveraging knowledge embedded in the world’s most resource-accessible language. We theorized that the availability of weak-parallel corpora such as multi-lingual Wikipedia articles acts as mediums for cross-lingual knowledge transfer (Hu et al., 2020). As shown in the results (Table 6), the base version of our model improved in reading comprehension and abstract summarization tasks but slightly suffered in classification and NLI tasks, suggesting that composition of knowledge-oriented data sources in the pretraining corpus may help the language model in tasks related to language generation (answer generation and summary generation) at the risk of slight underfitting in discriminatory power.

heading: A.2 Benchmark Datasets
text: This section provides more details on the benchmark datasets.
• NSMC|| is a movie review dataset constructed from NAVER Movie, a Korean movie review website, and consists of 150k training data and 50k test data samples, labeled with positive and negative classes.
• YNAT is a news-topic classification dataset as a part of the Korean Language Understanding Evaluation (KLUE) (Park et al., 2021) benchmark set. The dataset consists of titles for news articles
||https://github.com/e9t/nsmc
and the corresponding news topic labels. The dataset has 45.6k training data samples, 91k validation data samples, and 91k test data samples.
• KLUE-NLI is a natural language inference dataset from the KLUE benchmark set. Similar to MNLI (Williams et al., 2018), each sample in the dataset contains a pair of premise-hypothesis sentences, and the goal is to label the pair with one of "entailment", "neutral", or "contradiction". It comprises 25k training data, 3k validation data, and 3k test data samples.
• KLUE-STS is designed to evaluate a model’s ability to capture the semantic similarity between two sentences. Like YNAT and KLUE-NLI, KLUE-STS is also a part of the KLUE benchmark. The dataset consists of 11.6k training data, 519 validation data, and 1k test data samples.
• KorQuAD is a Korean question Answering dataset for machine reading comprehension (Lim et al., 2019), similar to SQuAD (Rajpurkar et al., 2016). The dataset consists of 60k question/answer pairs for training, 5.8k for validation, and 3.9k for testing.

heading: B Supplementary Materials Related to Dialog-Oriented Adaptation
text: This appendix section contains supplementary materials related to the dialog-oriented adaption of HyperT5.

heading: B.1 Dialog-oriented Adaptation Corpus
text: Here is the detailed list of data sources for constructing the dialog heavy-finetuning corpus presented (Table 7). The data consist of both open-sourced (Modu, AI Hub) and proprietary dialog corpus. Modu datasets are a collection of various dialog-oriented datasets collected by National Institute of Korean Language (NIKL)**.

heading: B.2 AI Hub Benchmark Datasets
text: This section provides more details on the dialog-oriented benchmark datasets. Note that the benchmark datasets are excluded from the dialog adaptation corpus.
• AI Hub ToD is a task-oriented dialog (ToD) dataset from AI Hub††, which covers 20 different topics (restaurant booking, online shopping QA, etc.). We preprocess the corpus to build 38.5k of training data and 3.9k of test data. We used the ToD dataset for dialog in-filling and dialog response
**https://corpus.korean.go.kr/ ††https://bit.ly/3S9Wxi6
generation. Each dialog session produces 3 or 4 training samples by random utterance selection. For the dialog response generation task, the turns before the selected utterance are only used for the dialog context, whereas both turns before and after the selected utterance are given as the context for dialog infilling. We also benchmarked the dialog topic classification using the ToD dataset with topic labels.
• AI Hub ODD is an open-domain dialog (ODD) dataset from AI Hub‡‡, over 20 different topics (social issues, food, marriage, etc.). We built a dataset with 87.7k training data and 11.0k of test data for the aforementioned tasks in AI Hub ToD. Similarly to AI Hub ToD, each dialog session results in multiple training samples.
• AI Hub Script-Summ is a broadcasting media transcript summarization dataset from AI Hub. We built a dataset with 84.4k training data and 10k test data for dialog summarization. Finally, we use the ROUGE score for generation task evaluation and Macro F1-Score for classification.

heading: B.3 Scarce Data Benchmark Results
text: Table 8 illustrates the scarce data benchmark results for our dialog-adapted models against HyperT5 models as baselines. We averaged the experimental results over five different random seeds. All the experiments are under the early stopping option with a patience level of 5. For each experiment, the best checkpoint is determined according to the evaluation metric. We set the learning rate to 5e-4 with linear learning decay.
Note that DialogHT5 shows a huge performance leap in DI tasks. This can be explained by the fact that dialog in-filling is essentially a single utterance masking (SUM), hence the MUM objective we used for dialog adaptation is a more challenging version of dialog in-filling.
In the extreme data-scarce settings (i.e., the training sample number of 0.1k), both HyperT5 and DialogHT5, regardless of the model size, tend to fail training without hyperparameter tuning on the learning rate. In general, using 5e-4 instead of 5e-5 enables tuning to begin working.

heading: B.4 Full Data Benchmark Results
text: We further conduct the full data benchmarks. Results show that DialogHT5 models achieve higher scores compared to HyperT5 models in most cases (Table 9).
‡‡https://bit.ly/3kc75R5 https://bit.ly/3Izjmsv

heading: Future Work
text: For future work, we wish to generalize the domain adaptation approach and study the efficacy of multi-task learning (Aribandi et al. , 2022) from the industrial perspective. Furthermore, we look forward to conducting comprehensive investigations into cross-architectural optimality. ¶Conversely, this means that decoder-only architectures scale better with larger compute budgets (Figure 3b)