: Korean Character Representations Based on the Combination Rules of Subcharacters (2024)

Abstract: The Korean writing system, Hangeul, has a unique character representation rigidly following the invention principles recorded in Hunminjeongeum.1 However, existing pretrained language models (PLMs) for Korean have overlooked these principles. In this paper, we introduce a novel framework for Korean PLMs called , which firstly brings the invention principles of Hangeul to represent character. Our proposed method, , exhibits notable experimental proficiency across diverse NLP tasks. In particular, our method outperforms the state-ofthe-art Korean PLM by an average of 2.11% in five Korean natural language understanding tasks. Furthermore, extensive experiments demonstrate that our proposed method is suitable for comprehending the linguistic features of the Korean language. Consequently, we shed light on the superiority of using subcharacters over the typical subword-based approach for Korean PLMs. Our code is available at: https://github.com/SungHo3268/KOMBO.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 5102–5119 August 11-16, 2024 ©2024 Association for Computational Linguistics
, exhibits notable experimental proficiency across diverse NLP tasks. In particular, our method outperforms the state-ofthe-art Korean PLM by an average of 2.11% in five Korean natural language understanding tasks. Furthermore, extensive experiments demonstrate that our proposed method is suitable for comprehending the linguistic features of the Korean language. Consequently, we shed light on the superiority of using subcharacters over the typical subword-based approach for Korean PLMs. Our code is available at: https://github.com/SungHo3268/KOMBO.

heading: 1 Introduction
text: Determining the representation of a word is the first stepping stone in building pre-trained language models. The predominant approach in English has attempted to decompose each word (e.g., pureness) into subwords (e.g., pure + ness) based on the frequency of the words, such as byte-pair encoding (BPE) (Sennrich et al., 2016), WordPiece (Schuster and Nakajima, 2012), and SentencePiece (Kudo and Richardson, 2018). Notably, even for the Korean language, which has a significantly different linguistic structure from English, the subwordbased approach has been widely adopted for Korean PLMs (Park et al., 2020, 2021).
∗These authors contributed equally to this work. 1Hunminjeongeum is a book published in 1446 that describes the principles of invention and usage of Hangeul, devised by King Sejong (National Hangeul Museum, 2018).
However, the Korean writing system (known as Hangeul) has a unique property in representing letters. Unlike English, which typically adheres to a "word-subword-character" structure, Korean includes an additional "subcharacter" level consisting of chosung (initial consonants), joongsung (vowels), and jongsung (final consonants). This leads to a "word-subword-character-subcharacter" structure, giving rise to distinct Korean linguistic features. This structure is even problematic when generating compound words, such as기찻 길train track.2 Despite these nuances in subcharacters with the same meaning, subword-based methods are blind to this information, typically parsing the compound word as separate tokens: 기,찻, and길. Such structural differences raise questions about the suitability of the subword-based method for the Korean PLMs, considering that it may overlook important linguistic information beyond characters.
In this work, given that it is crucial to ground the linguistic nuances and unique structures of the Korean language, we draw our attention towards subcharacter in building Korean PLMs. To accurately reflect the linguistic information of subcharacters, we bring the historical document Hunminjeongeum, which provides comprehensive insights into the design principles of subcharacters and their combination rules. The following are the two essential statements related to subcharacters (National Hangeul Museum, 2021):
• [Design of the Letters] Chosung (initial consonant), joongsung (vowel), and jongsung (final consonant) are combined to form a single character representing a syllable.
• [Combination of the Letters] Chosung can be placed above joongsung, or it can be positioned to the left of joongsung. Jongsung is placed below chosung and joongsung. 2기찻길train track is formed from기차train and길track, where a subcharacter ‘ㅅ’ is inserted between two words.
5102
Building on these principles, we propose a novel framework for Korean PLMs, referred to as (KOrean character representations based on the coMBinatiOn rules of subcharacters). To instill the design principles and combination rules of subcharacters into PLMs, starts with subcharacters as the initial representations. These representations are progressively combined to form a character through a merging layer guided by the combination principles. Additionally, we also introduce a masking strategy tailored to subcharacters, aimed at learning the structural knowledge of characters during pre-training. Consequently, such a comprehensive and fundamental approach enables PLMs to better comprehend the unique structure and composition of the Korean language.
To demonstrate the efficacy of the proposed method, we perform extensive experiments over a wide range of NLP tasks and compare ours with a variety of Korean tokenization methods in building PLMs. The results convincingly show that considering subcharacter, especially for Jamo, in a principled manner brings substantial improvement to Korean PLMs. Moreover, the in-depth analysis supports that can better understand the Korean features, such as conjugations and offensive language, compared to existing methods. In summary, the contributions of this paper include the following:
• We present , a novel framework for Korean PLMs grounded in the invention principles of Hangeul as specified in Hunminjeongeum.
• We integrate the design of characters and combination rules into neural language models, marking a novel exploration in Korean PLMs.
• We demonstrate that considering the structure of Hangeul through invention principles leads to remarkable performance on a wide range of NLP tasks, thereby shedding light on the potential of Jamo units for Korean PLMs.

heading: 2 Related Work

heading: 2.1 Korean Pre-trained Language Models
text: Pre-trained language models have understood the word by splitting it into various atomic units. Similar to other languages, Korean PLMs have also adopted tokenizing sentences into character-level (Cho et al., 2019), subword-level (Lee et al., 2018; Park et al., 2018), or have split by whitespace (Eo
et al., 2022). Moreover, to consider the morphologically rich feature of the Korean language, Lee and Shin (2021) and Moon et al. (2022) have utilized a morpheme analyzer (Kudo, 2005) to tokenize sentences. Park et al. (2020), Kim et al. (2021), and Park et al. (2021) have presented morpheme-aware subword tokenization, which is the subword-level tokenization approach applying a morpheme analyzer before implementing BPE. Within this diverse range of units, most of the current Korean PLMs have used morpheme-aware subword units as basic tokens to split the sentence because of their high performance. However, subword-level methodologies have been insufficient in addressing the distinctive linguistic attributes (Albright and Kang, 2009; Park and Shin, 2018) inherent in the Korean language, which are influenced by the presence of subcharacters specific to Hangeul.

heading: 2.2 Korean Subcharacter Units
text: To consider the unique structural information of Korean letters, there have been some trials to tokenize the word into subcharacters for word embeddings. Stratos (2017) and Moon and Okazaki (2020) have split words into the subcharacter unit Jamo, which is used as a basic subcharacter of Hangeul letters. Additionally, Kim et al. (2022) have decomposed Jamo into smaller tokens, BTS units, which are inspired by the invention principle of Hangeul. Depending on the decomposition level, they have discerned BTS units into three different units, such as Stroke, Cji, and BTS. However, most of the previous works have been exploited only in static word embeddings, leaving a research gap in the application in PLMs, such as BERT (Devlin et al., 2019). Therefore, in this work, we propose a novel framework for Korean PLMs based on the design principles and combination rules of subcharacters as mentioned in Hunminjeongeum.
3
In this section, we elaborate in details for Korean PLMs. The framework begins with taking subcharacters as initial representations (§3.1). These subcharacter representations are progressively combined to form a character based on the combination rules (§3.2). After passing through the transformer blocks (§3.3), the subcharacter representations are reconstructed to perform token-level objectives (§3.4). To learn the structural knowledge of subcharacters during pre-training, we also
introduce the span-subcharacter masking strategy (§3.5). Figure 1 illustrates the overall procedures.

heading: 3.1 Initial Representations of
text: begins with treating subcharacters as atomic units for representing a word. While there are a number of ways to represent subcharacters (e.g., Stroke, Cji, BTS), we primarily use Jamo in the following sections for the sake of simplicity in explanation.3
Subcharacter Tokenization Each input character is decomposed into chosung, joongsung, and jongsung. Chosung and joongsung are essential components to create a character, whereas jongsung is an optional component. To represent the absence of jongsung, we use the special empty token ( ), ensuring that each character is always represented with three Jamo units (e.g., a character ‘차car’ is decomposed to ㅊ,ㅏ, and ). We denote the subcharacter input sequence as x = {x1, x2, · · · , xN} ∈ RN , where N is the number of subcharacters.
Subcharacter Embedding We project the input sequence x onto the embedding space. The pro-
3Details regarding BTS units are provided in Appendix A.
jected token representations are denoted as follows:
e = Embedding(x) ∈ RN×D (1) where D is the dimension of embeddings.

heading: 3.2 Subcharacter Combination
text: Contextualization Building on the principles of combination, we integrate subcharacter representations to construct a character. However, simply merging these representations without considering context and sequence fails to capture the intricate composition of characters. Therefore, we apply contextualization to the subcharacters using shallow local transformer blocks, subsequently followed by Gated Recurrent Units (GRU) (Cho et al., 2014). Through the contextualization layers, the subcharacter representations are derived as follows:
h = GRU(LocalTransformer(e)) ∈ RN×D (2) Merging Jamo units In the following sections for the intuitive illustration, we refer to the structural representation hi ∈ h as chosung hI,k, joongsung hV,k, and jongsung hF,k, respectively:
hi =    hI,k if i = 3k − 2 hV,k if i = 3k − 1 hF,k if i = 3k
(3)
where each k ∈ [1, N/3] corresponds to a character index in the sequence.
Subsequently, to form a character representation, we combine denoted subcharacters in two steps. Depending on the order of combining subcharacters as referred to Design of the Letters, we first merge chosung and joongsung through elementwise summation of hI = {hI,k} and hV = {hV,k}.
hI+V = hI + hV ∈ R N 3 ×D (4)
The second merging step is to combine the resultant representations with jongsung. Considering that jongsung is always positioned below chosung and joongsung, as mentioned in Combination of the Letters, we treat it by performing vertical concatenation of jongsung representations hF = {hF,k} with the combined chosung and joongsung representations hI+V. Formally, this can be represented as follows:
hR = [ hI+V hF ] ∈ R2×N3 ×D (5)
To generate the final subcharacter jongsung based on the merged representations, we perform convolution and pooling operations4 over the concatenated representations hR.
hC = AvgPool(Conv(hR)) ∈ R N 3 ×D (6)
This results in a dense character representation grounded in the combination rules of subcharacters.

heading: 3.3 Transformer Stack
text: On top of the merged representations, we employ a series of transformer, consisting of L layers, to achieve contextualization as follows:
h′C = TransformerL(hC) (7)
The configuration of these transformer layers follows the same design as if the BERT model (Devlin et al., 2019). Notably, it is crucial to highlight that our framework is primarily designed to manipulate token representations, allowing for seamless integration with various transformer-based architectures, extending beyond BERT.
4We heuristically explore and use (2× 1) kernels with the stride of one to perform convolution.

heading: 3.4 Subcharacter Restoration
text: In sentence-level tasks, the first token (i.e., [CLS]) from the transformer stack is utilized to perform the specified task. However, token-level classification necessitates a fine-grained sequential output that aligns with the vocabulary, requiring subcharacter representations. We thus introduce the restoration layers after the transformer stack, which convert the character representations back into the constituent subcharacters.
The reconstruction proceeds by reversing the process of the subcharacter combination. First, the character representations (h′C or h ′ R) is duplicated by the number of tokens used in each subcharacter combination process (Eq. (6) or Eq. (4)). Inspired by U-Net (Ronneberger et al., 2015), the duplicated representations are subsequently combined with the original subcharacter representations (i.e., hR or h) for better restoration. Finally, by leveraging the GRU layer, we ensure the continuity between subcharacters during the reconstruction process. In summary, the restoration process5 can be formulated as follows:
h′R = GRU(Repeat(h ′ C) + hR) ∈ R 2 3 N×D (8)
h′ = GRU(Repeat(h′R) + h) ∈ RN×D (9)

heading: 3.5 Span-Subcharacter Masking
text: Furthermore, to learn the linguistic structure within characters and their corresponding subcharacters, we also introduce a span-subcharacter masking strategy inspired by SpanBERT (Joshi et al., 2020). Specifically, instead of masking tokens at arbitrary positions, we mask out the consecutive three subcharacters (corresponding to chosung, joongsung, and jongsung) of each character for the objective of MLM. Such a masking strategy encourages the model to learn the compositional relationships between subcharacters in the pre-training phase, thereby leading to subcharacter-aware pre-trained language models.

heading: 4 Experiments
text: In this section, we conduct a comparative analysis between the proposed method and existing Korean tokenization models on both standard Korean datasets and noisy Korean datasets.
5Comparison with the variation of restoration process is provided in Appendix B.1.

heading: 4.1 Experimental Setup
text: Models We compare our proposed methods with Korean PLMs using 9 distinct tokenization methods, including BTS units (Stroke, Cji, and BTS), Jamo, Character, Morpheme, Subword, MorSubword (short for Morpheme-aware Subword (Park et al., 2020, 2021; Kim et al., 2021)), and Word units. We uniformly apply BERT to all PLMs, using the same configuration as BERTbase with 12 transformer blocks. Depending on the types of subcharacters, we categorize our methods as Stroke, Cji, BTS, and Jamo. We match the size of all our models with the state-of-the-art Korean PLM, MorSubword, by adjusting the number of local transformer blocks in the contextualization layer.
Pre-training We pre-trained all models for 1M steps on Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) tasks as BERT. We used the Korean Wiki corpus and the Namuwiki corpus, 6.2 GB, including about 46M sentences in total. The details of data preprocessing and hyperparameter settings are explained in Appendix C.
Evaluation Pre-trained models were individually fine-tuned on each downstream task dataset. We report all experimental results as the average values obtained from three random seeds. To verify the effectiveness of the proposed method in stan-
dard Korean dataset, we evaluated the models on five Korean NLU tasks, including machine reading comprehension (KorQuAD 1.0), natural language inference (KorNLI), semantic textual similarity (KorSTS), sentiment analysis (NSMC), and paraphrase identification (PAWS-X). We provide detailed explanations about the overall data information and hyperparameter settings in Appendix D.1.
Moreover, to assess the robustness of models in noisy Korean settings, we synthetically injected the typos into the Korean NLU datasets, such as KorNLI, KorSTS, NSMC, and PAWS-X. We randomly generated typos using four different typo methods, following Zhuang and Zuccon (2021); Insertion: randomly add a letter which is adjacent of the letter on the keyboard, Transposition: randomly switches a letter with one of its neighbor letter, Substitution: randomly changes a letter with one of its neighbor letters on the keyboard, and Deletion: drops a random letter. The other settings, unless specified, follow the same manner as Korean NLU tasks.

heading: 4.2 Experiment Results

heading: 4.2.1 Standard Korean Datasets
text: In Table 1, our proposed method consistently outperforms BERT models, which use the same subcharacter tokenization as the initial representation, for all tasks with marginal increases,
averaging more than 9%. We observe that, among ours, Jamo presents the most overwhelming performances, while for BTS units shows lagging performances. We suspect that this is due to the truncated initial inputs at subcharacter level by a setting of maximum sequence length, giving rise to short context. Jamo outperforms the state-of-the-art model MorSubword across all tasks except for NSMC. Despite using only 0.53% of the size of the static embeddings compared to the MorSubword model, our method achieves higher performance by an average of 2.11%. These admirable results provide valuable insight into the prospective potential of Jamo, taking over the trend of current subword-based Korean PLMs.

heading: 4.2.2 Noisy Korean Datasets
text: We evaluate the robustness of models to typos in two experimental settings. The first is the random typo setting, where we raise the typos by randomly selecting typo methods among Insertion, Transposition, Substitution, and Deletion, starting from the rate of 0% and increasing it by 5% up to 40%. In Figure 2, the baselines using larger token units, such as Subword and MorSubword, are highly vulnerable to typo, whereas the models using smaller token units, such as Jamo and Character, generally exhibit strong robustness with a gradual decline. Moreover, aligning with the trends shown in the baselines, our proposed model Jamo also exhibited strong robustness to typo. Furthermore, we can observe that the performance gap between Jamo and the state-of-the-art model is getting wider as if the typo rate increases, again demonstrating the strong robustness of our proposed method. The second setting is that we inject only one type of typo error among four different typo methods. In Table 2, we observe that our method is more powerful in the substitution typo method, which is the most similar setting to real-
world typographical errors (Jeon et al., 2010). We report more results for other tasks in Appendix D.2.

heading: 5 Ablations
text: In Table 3, we present ablation experiments on the subcharacter combination methods of . The subsequent segments present the experimental results corresponding to the modifications in the structure of subcharacter combination and kernel size used in merging jongsung, respectively. In Table 4, to investigate the effect of Korean character structural knowledge, we compare our subcharacter combination method with other downsampling methods generally used in English.

heading: 5.1 Impact of Contextualization
text: We evaluate our proposed model by removing the contextualization layer before merging embeddings. The omission of the contextualization layer has resulted with a significant drop in performance, which clearly indicates the assisting role of contextualization in character representations. Through these results, we insist that complementing small
Condition KorNLI KorSTS PAWS-X Total
Dev Test Dev Test Dev Test Avg
Jamo 74.22 73.77 84.47 77.43 73.66 70.88 75.74
Subcharacter Combination
w/o Contextualization 52.70 54.28 73.11 67.57 54.76 54.77 59.53 w/o Merging Subcharacters 71.45 71.68 82.54 76.27 71.97 68.42 73.72 w/o Addition of Jongsung 73.40 73.69 84.17 75.94 73.17 70.72 75.18 w/o Span-Subcharacter Masking 72.29 73.08 83.58 76.13 72.07 70.21 74.56
Kernel Size in Addition of Jongsung
w/ (2x2) kernel 73.70 75.38 83.04 75.40 65.07 63.13 72.62 w/ (2x3) kernel 73.31 74.68 82.62 75.88 62.73 61.34 71.76 w/ (2x1)+(2x2) kernel 73.78 74.84 82.95 75.27 64.46 62.17 72.25
Table 3: Ablation results for various components of Jamo. The first row is the best setting of Jamo, which merges all subcharacters sequentially. The best and second-best results are highlighted in boldface and underline, respectively.
Method KorNLI KorSTS PAWS-X Total
Dev Test Dev Test Dev Test Avg
Jamo 74.22 73.77 84.47 77.43 73.66 70.88 75.74
Attention Pooling* (Dai et al., 2020) 63.91 63.38 80.58 72.43 54.93 55.37 65.10 Linear Pooling* (Nawrot et al., 2022) 63.93 62.59 80.65 72.02 55.52 55.97 65.11
embeddings is necessary to enhance the performance of subcharacter models.

heading: 5.2 Impact of Span-Subcharacter Masking
text: Instead of applying the span-subcharacter masking strategy for MLM, we adopt the token-level, i.e., subcharacter masking strategy. We observe that using the span-subcharacter masking strategy enhances the quality of the combined character representation while increasing the average performance by 1%. Through the results, we verify the efficacy of span-subcharacter masking to learn the structural information of characters.

heading: 5.3 Impact of Merging Subcharacters
text: The model that merges subcharacters shows a significant performance improvement, averaging approximately 2.02% higher than the model without subcharacter merging. Furthermore, removing the addition of jongsung results in a performance decrease, averaging around 0.56%. These consis-
tently lower performances underscore the importance of the subcharacter merging steps.

heading: 5.4 Impact of Kernel Size
text: By changing the size and number of the kernels, we explore the most effective kernel size in our subcharacter combination method. The experimental results show that using a (2x1) size kernel, which looks at each character individually, yields the best performance across all tasks except the KorNLI test set. This indicates that the best understanding of Korean is obtained when focusing on the character level, which is the composite unit of Hangeul.

heading: 5.5 Impact of Korean Character Structure
text: In Table 4, to explore the effect of recognition of character structure in Korean, we compare our method with other English downsampling methods (Dai et al., 2020; Nawrot et al., 2022), which do not consider the unique Korean character structure. Instead of our subcharacter combination method,
we opt to employ each downsampling method to combine subcharacter embeddings. Both methods show significant performance drops by an average of over 10% compared to our method. We demonstrate that considering the structure of Hangeul is a keystone to comprehending the Korean language.

heading: 6 Analysis
text: In this section, we analyze two Korean linguistic characteristics. The primary objective is to investigate the resilience of predicates against variations arising from Korean conjugations, while the secondary aim involves evaluating the model’s robustness to Korean offensive languages.

heading: 6.1 Robustness to Character Conjugations
text: Korean, as an agglutinative language, presents a multitude of conjugational variations occurring at both the character and subcharacter levels through inserting, deletion, or substitution of endings, resulting in alterations in parts-of-speech (POS) or tense. For instance, the verb ‘차다 be cold’, composed of the stem ‘차cold’ and the ending ‘다 be’, can transform into the adjective ‘찬cold’ by substituting the ending ‘다 be’ with the subcharacter ‘ㄴ’. Consequently, a profound understanding of the subcharacter structure within Hangeul is essential for proficiently handling Korean conjugations. We compare the vanilla character model with Jamo, as illustrated in Figure 3. For the given target sentence, we extract embeddings for the characters ‘찬’ and ‘차’ from both models. Subsequently, we compute cosine similarities between these two embeddings and all character embeddings in the target sentence, visualizing the results on a heatmap. The vanilla character model with static embeddings distinguishes ‘찬cold’ and
‘차cold’ as distinct entities, whereas Jamo recognizes them as semantically similar characters, displaying significantly higher similarity scores for both embeddings (0.68) compared to the character model’s score of 0.21. Further instances are provided in Appendix E.

heading: 6.2 Robustness to Korean Offensive Language
text: Warning: This section contains several offensive statements.
To evaluate the robustness of our proposed method to offensive language, we experiment on the Korean offensive language datasets BEEP! (Moon et al., 2020), K-MHaS (Lee et al., 2022a), and KOLD (Jeong et al., 2022). Details for data and experimental settings are presented in Appendix D.3. In Table 5, Jamo shows the highest macro F1 score across all tasks, demonstrating its robustness on Korean offensive wordings. Interestingly, we find that the subword-based models struggle with the Korean offensive wordings; for example, they tokenize the compound word ‘개새끼puppy’, which is combined ‘개dog’ and ‘새끼pup’, into independent three characters ‘개’, ‘새’, and ‘끼’. 6

heading: 7 Conclusion
text: Although there is crucial linguistic information and clear invention principles in Korean letters, the existing Korean PLMs have overlooked them and just opted to use subword-level tokenization methods due to their high performance. In this paper, we first bring attention to the overlooked design principles and combination rules of subcharacters, as specified in Hunminjeongeum. We have introduced a novel framework for Korean PLMs called
6In Korean, ‘개새끼puppy’ is used as an insult to refer to someone in a disrespectful manner.
, which generates character representations following the design and combination rules of subcharacters. We have demonstrated the efficacy of on both standard Korean datasets and noisy Korean datasets by outperforming various Korean tokenization baselines on most of the tasks we evaluated. Additionally, through diverse ablations and analyses, we have shown that our proposed method improves the quality of character representations and has a more robust adaptability to Korean conjugations and offensive language. These convincing outcomes of Jamo following the invention principles of Hangeul can thereby provide an inspiring insight for the prospective potential of Jamo unit applying in Korean PLMs.
Limitations
While our proposed methodology demonstrates its suitability for Korean natural language processing, it comes with some limitations for future research. Although our proposed is designed with an encoder transformer model, our framework is primarily designed to manipulate token representations, allowing for seamless integration with various transformer-based architectures, extending beyond Encoder Models. Therefore, we believe that its dynamic ability to generate appropriate character embeddings for each input also works well in generative modeling, too. We leave the exploration of its value in generative models for future research.
In this work, we limit our focus to Korean representations up to the character-level, as our main objective is to incorporate Korean subcharacter units into PLMs following the principles of Hangeul. However, given the morphologically intricate nature of the Korean language, considering representations up to the subword-level is also impera-
tive. With our hierarchical combination approach demonstrating superior performance at the character level, we anticipate that it may serve as a precursor for future methodologies transitioning from character-level to subword-level representations in Korean language processing.
Ethics Statement
We only use three previously collected or synthetically generated Korean offensive language benchmarks, which are annotated with humans who have verified their qualifications. We strictly follow the data usage agreements for each public dataset we implement in this paper. We also mention a warning statement in §6.2, where the offensive statements are directly used in this paper. Additionally, we recognize the potential that the high performance in Korean offensive language also means the model might be biased toward misused or offensive language. However, we believe that the high robustness of our method is aligned with the advantages due to the deep comprehension of Korean character structure, not the familiarity with toxic wording.

heading: Acknowledgements
text: This work was supported by the Basic Research Program through the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2021R1A2C3010430) and Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.RS-2019II190079, Artificial Intelligence Graduate School Program (Korea University)).

heading: A for BTS units
text: Kim et al. (2022) introduced Basic, Tiniest Subword (BTS) units for the Korean language, inspired by the invention principle of Hangeul. BTS units comprise 5 basic consonants and 3 basic vowels, defined as atomic units in Hunminjeongeum. The decomposition of Korean characters into BTS units is referred to as BTS decomposition, where the consonant is split into a maximum of 4 subcharacters (e.g., a consonant ‘ㅋ’ is decomposed into ‘ㄱ’ and ‘-’), and the vowel is split into a maximum of 5 subcharacters (e.g., a vowel ‘ㅏ’ is decomposed into ‘l’ and ‘·’). Additionally, there are two variant decomposition units: the consonantonly BTS decomposition (denoted as Stroke) and the vowel-only BTS decomposition (denoted as Cji, short for Cheonjiin). In this paper, we consider all applications for BTS units as subcharacter units and present as Stroke, Cji, and BTS. To simplify, we collectively call these three models as BTS units. We illustrate Jamo in Figure 4, as a representative example among BTS units.
Subcharacter Tokenization Unlike Jamo units, which have three subcharacters for each character, BTS units decompose Jamo into smaller subcharacters. Depending on the type of BTS unit, the number of tokens comprising one character is different. Stroke has 9, Cji has 7, and BTS has 13 tokens for each character. To ensure that the number of tokens forming each Jamo is always maximum, we use the special empty token ( ).
Subcharacter Embedding By the same process in Eq. (1), we project the subcharacter sequence onto the embedding space, denoted as:
e = {e1, e2, · · · , eN} ∈ RN×D (10)
where N is the number of subcharacters, and D is the dimension of embeddings.
Contextualization To capture the intricate composition of characters, we apply contextualization to the subcharacters, same as in Eq. (2).
h = GRU(LocalTransformer(e)) ∈ RN×D (11)
Merging Jamo units The contextualized subcharacter representations are combined to form a character representation similar to §3.2. Before
merging chosung, joongsung, and jongsung, there is a preceding step combining the subcharacter representations into the representations of chosung, joongsung, and jongsung. We primarily use Stroke (consonant-only BTS decomposition) in the following sections for the sake of simplicity in explanation. For each k ∈ [1, N/9], hI,k, hV,k, and hF,k in Eq. (3) can be denoted as follows:
hI,k = 4∑
j=1
h9(k−1)+j (12)
hV,k =
5∑
j=5
h9(k−1)+j (13)
hF,k = 9∑
j=6
h9(k−1)+j (14)
Then, we sum the combined chosung and joongsung representations to form the intermediate representation hI+V with the same process as in Eq. (4).
hI+V = hI + hV ∈ R N 9 ×D (15)
To consider the position of jongsung, below chosung and joongsung, we vertically concatenate the combined representations hI+V and jongsung hF, represented in Eq. (5).
hR = [ hI+V hF ] ∈ R2×N9 ×D (16)
The resultant representations are then performed using the operations in Eq. (6) to generate the character representation.
hC = AvgPool(Conv(hR)) ∈ R N 9 ×D (17)
Subcharacter Restoration We also reconstruct the outputs of the transformer stack into subcharacters. Due to the simplicity of the addition of subcharacters into Jamo, we directly decompose intermediate representations into BTS units, jumping the decomposition step from Jamo to BTS units. Thus, the process of subcharacter restoration is almost the same as in Eq. (8) and Eq. (9), but different in the size of the vectors. More formally,
h′R = GRU(Repeat(h ′ C) + hR) ∈ R 2N 9 ×D (18)
h′ = GRU(Repeat(h′R) + h) ∈ RN×D (19)

heading: B Additional Ablations
text: B.1 Impact of Subcharacter Restoration
We explore the effects of the various subcharacter restoration methods. Table 6 shows the experimental results. We observe that leveraging only a single GRU layer does not help subcharacter reconstruction. However, the GRU layers in hierarchical manners are effective for increasing the granularity of character-level sequences. Moreover, the residual connection, which combines duplicated representations with the original subcharacter embeddings, significantly enhances the performance when combined with the GRU layer, whereas it does not provide any advantages in the case of the linear layer.

heading: C Details of Pre-training
text: Data Preprocessing We use the Korean Wiki corpus and the Namuwiki corpus, extracted from
dump data using data extractors such as WikiExtractor 7 and NamuWikiExtractor 8. After applying the data extractor to dump data, we clean the data by removing irregular empty spaces and parsing traces, such as HTML tags. We only retain Korean and English text and punctuation with regular expression. As a result, we can get a total of 6.2 GB, including about 46 million sentences.
Training Settings Each model is pre-trained for 1M steps with a batch size of 128 using the RTX 3090 GPU. We set the AdamW optimizer (Loshchilov and Hutter, 2019) with a learning rate of 5e-05 warm-up over the first 10K steps. We use a sequence length of 128. To match the size of our models with the state-of-the-art Korean pre-trained language model, MorSubword, we adjust the number of local transformer blocks in contextualization layers. The number of parameters in both mod-
7WikiExtractor 8NamuWikiExtractor
els is 110M, but the training time of Jamo is almost half of vanilla MorSubword model. We compare the number of parameters and the training time rate in Figure 5.

heading: D Details of Fine-tuning
text: D.1 Standard Korean Datasets
Datasets KorQuAD 1.0 (Lim et al., 2019) is the Korean version of the well-known SQuAD 1.0 dataset (Rajpurkar et al., 2016), designed for the machine reading comprehension task. KorQuAD 1.0 dataset consists of 10,645 passages and 66,181 question-answer pairs (60,407 for the training set and 5,774 for the development set). Similar to SQuAD 1.0 dataset, the evaluation involves the model predicting the position of the answer within the given passage.
KorNLI (Ham et al., 2020) is a dataset consisting of 942,854 training sets and 7,500 evaluation
sets for natural language inference, sourced from SNLI dataset (Bowman et al., 2015), MNLI dataset (Williams et al., 2018), and XNLI dataset (Conneau et al., 2018). The labels of the dataset are entailment, contradiction, and neutral.
KorSTS (Ham et al., 2020) is designed to evaluate the semantic similarity between two sentences. It is a Korean adaptation of STS-B dataset (Cer et al., 2017). KorSTS dataset comprises 5,749 training examples and 2,879 evaluation examples. Each sample is labeled with a similarity score scaled from 0 to 5, indicating the similarity between the two sentences.
NSMC (Park, 2016) is a NAVER movie review dataset used for performing a sentiment analysis of each Korean sentence. NSMC dataset consists of 150,000 training samples and 50,000 test samples, with each sentence labeled as negative or positive.
PAWS-X (Yang et al., 2019) is a dataset for the paraphrase identification task. PAWS-X dataset has six language tasks, and we only evaluate models on the Korean subset. It consists of 53,338 sentence pairs (49,410 for the training set, 1,965 for the development set, and 1,972 for the test set), and each label has two possible values, different meanings, or paraphrases.
Data Preprocessing To focus solely on evaluating Korean language processing capabilities, we convert English words into Hangeul based on the International Phonetic Alphabet (IPA) symbols 9 (e.g., ‘bus’ → ‘버스’) and remove special characters.
Hyperparameters For each of the datasets, we use the following hyperparameters: KorQuAD 1.0 uses 5 epochs and a batch size of 16 . KorNLI uses 3 epochs and 16 batch size. KorSTS uses 5 epochs and a batch size of 64. NSMC uses 3 epochs and a batch size of 64. PAWS-X uses 5 epochs and a batch size of 64. We select the best learning rate (among 1e-04, 4e-05, and 5e-05) on the Dev set and warmup over the first 10% steps of the total. We use a dropout probability of 0.1. We choose the maximum sequence length (among 128, 256, 512, 1024, and 2048) depending on the type of token.
D.2 Noisy Korean Datasets
We conduct additional typo experiments for the KorSTS, NSMC, and PAWS-X datasets. In Figure 6,
9To convert graphemes to phonemes, we leverage the g2pK library. Refer to https://github.com/Kyubyong/g2pK
we illustrate the result of random typo experiments. In Table 7, we present the results of each of the four distinct typo generation methods.
D.3 Korean Offensive Language Dataset Datasets BEEP! (Moon et al., 2020) is a Korean corpus annotated for toxic speech detection, which consists of 10K manually human-annotated Korean corpus collected from a Korean entertainment news aggregation platform. The labels include Hate, Offensive, and None, categorizing the aggressiveness of the given sentences. To distinguish whether the text contains offensive language or not, we reformulate the labels into ‘Hate or Offensive’ and ‘None’ for binary classification.
K-MHaS (Lee et al., 2022b) is a Korean multilabel hate speech detection dataset that consists of 109K utterances from Korean news comments. KMHaS requires the classification of fine-grained discrimination labels among Politics, Origin, Physical, Age, Gender, Religion, Race, and Profanity based on the input sentences.
KOLD (Jeong et al., 2022) is a dataset for detecting the Korean offensive language. KOLD comprises 40,429 comments, annotated hierarchically with the type and the target. KOLD has three tasks: Level A (Offensive language detection), Level B (Target types categorization), and Level C (Target group Identification). In this paper, we implement the most difficult task, Level C in KOLD, labeling into 22 target groups: LGBTQ+, Men, Women, Asian, Black, Chinese, Indian, Korean-Chinese, Southeast Asian, White, Conservative, Progressive, Buddhism, Catholic, Christian, Islam, Agism, Disabled, Diseased, Feminist, Physical Appearance, and Socio-economic Status.
For all offensive language datasets, we convert English words into Hangeul based on the International Phonetic Alphabet (IPA) symbols.
Hyperparameters We train the models on BEEP! for 10 epochs with 32 batch size. In the case of the K-MHaS dataset, we train the models for 4 epochs with 32 batch size. For the experiments on KOLD, we train the models for 5 epochs with a batch size of 32 using a learning rate of 5e05. We choose the max sequence length (among 128, 256, 512) depending on the type of tokens.

heading: E Robustness to Character Conjugation
text: We illustrate more examples about Korean character conjugations in Figure 7, 8, 9, and 10.

heading: Future Work
text: Limitations
While our proposed methodology demonstrates its suitability for Korean natural language processing, it comes with some limitations for future research. Although our proposed is designed with an encoder transformer model, our framework is primarily designed to manipulate token representations, allowing for seamless integration with various transformer-based architectures, extending beyond Encoder Models. Therefore, we believe that its dynamic ability to generate appropriate character embeddings for each input also works well in generative modeling, too. We leave the exploration of its value in generative models for future research. In this work, we limit our focus to Korean representations up to the character-level, as our main objective is to incorporate Korean subcharacter units into PLMs following the principles of Hangeul. However, given the morphologically intricate nature of the Korean language, considering representations up to the subword-level is also impera-
tive. With our hierarchical combination approach demonstrating superior performance at the character level, we anticipate that it may serve as a precursor for future methodologies transitioning from character-level to subword-level representations in Korean language processing