Length-aware Byte Pair Encoding for Mitigating Over-segmentation in Korean Machine Translation (2024)

Abstract: Byte Pair Encoding is an effective approach in machine translation across several languages. However, our analysis indicates that BPE is prone to over-segmentation in the morphologically rich language, Korean, which can erode word semantics and lead to semantic confusion during training. This semantic confusion, stemming from over-segmentation, ultimately contributes to a degradation of overall translation quality. To address this issue, we introduce Length-aware Subword Vocabulary Construction (LeVoC), a novel approach strategically incorporating longer words into the vocabulary. By utilizing an external monolingual Korean corpus, LeVoC extracts and integrates long words, effectively preserving morphological information and reducing semantic confusion. Our experiments demonstrate that LeVoC not only significantly outperforms BPE, but also can be applied to and surpass current state-of-the-art morpheme-aware subword tokenization methods. We provide evidence that the difficulty in translating sentences with long words in Korean is associated with morphological compositionality, and LeVoC’s ability to reduce semantic confusion during training leads to improved translation quality.

heading: 1 Introduction
text: The utilization of subword-unit segmentation methods has become a prevailing trend in neural machine translation (NMT), with widespread acceptance among researchers (Farhad et al., 2021; Barrault et al., 2020). Among the various methods available, Byte Pair Encoding (BPE) (Sennrich et al., 2016) has emerged as a popular choice in numerous NMT studies due to its language-agnostic applicability and its ability to strike a balance between compression efficiency and computational complexity (Vaswani et al., 2017; Ng et al., 2019;
∗Equal Contribution † Corresponding Author
Zhou et al., 2022; Dankers et al., 2022; Zhang et al., 2022; Zhang and Feng, 2022; He et al., 2022).
Although BPE is effective, it often results in over-segmentation when applied to Korean, an agglutinative and morphologically rich language (Keren et al., 2022). This issue primarily stems from the language-agnostic and frequency-based characteristics of BPE, which can inadvertently lead to excessive segmentation of words.
Concretely, we identified three potential instances of over-segmentation, as presented in Table 1. The first example is when the subword created via segmentation loses its meaning. For example, when the Korean word “우두머리 (Chief Leader)” is segmented into “우두 +머리”, “우두” becomes a word that does not have any meaning by itself. This results in creating a subword of null meaning that cannot be applied in any context, such as a prefix or suffix. In the second example, the word’s meaning may not be preserved even when the individual meanings of subwords are combined. For instance, the meaning of “큰코다치다 (pay dearly)” changes when it is interpreted as a combination of the subwords “큰 (big)” + “코 (nose)” + “다치다 (hurt).” This demonstrates that the word’s meaning cannot be fully preserved simply by segmenting it into interpretable units. The third example pertains to the increase in semantic ambiguity of subwords caused by word segmentation. For instance, consider the compound word “바람개비 (Windmill).” This word can be segmented as “바람 (wind or affair) + 개비 (whirligig)” since it is a
2287
compound word. However, the subword “바람” itself can have multiple meanings, such as "wind" or "affair," which results in ambiguity after segmentation.
Over-segmentation not only distorts the meaning of sentences but also introduces semantic noise during the training phase, which can significantly degrade the translation quality (Keren et al., 2022). In our investigation, we have observed that this semantic noise is further exacerbated in domain shift scenarios, where the model is challenged with outof-domain test sets. This accumulation of semantic confusion is particularly detrimental in translation tasks where the fidelity of sentence semantics is paramount. Our findings underscore the critical need for addressing over-segmentation to ensure robustness against domain shifts and to maintain the integrity of translated content.
To address this challenge, we propose LeVoC (Length-aware Subword Vocabulary Construction), a novel approach specifically designed to mitigate the over-segmentation issue in Korean. By prioritizing morphologically rich long words during tokenization, LeVoC effectively alleviates oversegmentation, paving the way for more accurate and meaningful translations. To implement LeVoC, we first create a large-scale vocabulary using BPE and then build an auxiliary vocabulary set consisting of morphologically rich long words derived from it. We then construct a new tokenizer by incorporating the auxiliary vocabulary into the smallsized BPE vocabulary. More specifically, LeVoC preserves the existing tokenizer’s capacity to overcome the issue of out-of-vocabulary (OOV) words, and the integration of long words from the auxiliary vocabulary additionally relieves the tendency for over-segmentation, leading to a more coherent and accurate representation. Through our experiments, we demonstrate that LeVoC mitigates over-segmentation issues and reduces semantic confusion in Korean NMT tasks, thereby enhancing translation performance.
We demonstrate that the NMT model trained using LeVoC effectively preserves the morphological structure of the sentences, resulting in superior performance on out-of-domain data that were not present during training. To the best of our knowledge, our method is the first to enhance translation performance by addressing over-segmentation issues stemming from the linguistic characteristics of Korean in an unsupervised manner. We publicly
release LeVoC to support further studies1.

heading: 2 Related Works
text: The constraints of BPE (Sennrich et al., 2016), especially when applied to agglutinative languages with rich morphological features, have been extensively reported. These languages, characterized by the fusion of numerous morphemes into individual words, pose a significant obstacle to the subword tokenization approach, which is fundamentally based on frequency or likelihood of occurrence (Bostrom and Durrett, 2020; Nzeyimana and Rubungo, 2022).
Several studies have attempted to address these limitations. For instance, some studies have applied morphological segmentation prior to subword tokenization (Park et al., 2019, 2020), while others have utilized Part-of-Speech (POS) tagging as additional tag features (Chimalamarri and Sitaram, 2021). Hofmann et al. (2022) proposed a method to preserve the morphological structure by recursively exploring the longest sub-string.
In addition, some studies have explored alternative approaches to subword tokenization. Clark et al. (2022) introduced a tokenization-free deep encoder that allows parameters to be shared by hashable code points without assigning embeddings for each character. Aguilar et al. (2021) proposed a method to handle noises robustly by learning and generating the subword embeddings through a pretrained language model without any restrictions on the vocabulary.
While numerous studies have explored subword tokenization, the issue of over-segmentation specific to Korean language processing remains largely unexplored. LeVoC mitigates over-segmentation through a focus on vocabulary construction particularly in the context of the Korean language.

heading: 3 Proposed Method

heading: 3.1 Length-aware Subword Vocabulary Construction
text: Motivation In agglutinative languages, words are composed of several interdependent morphemes. This characteristic can be seen as a positive aspect in that it allows for constructing a vocabulary that covers most of the words in the dictionary using only a small number of subwords. This is the ideal scenario, wherein a small-sized vocabulary
1The code for LeVoC is publicly available at https:// github.com/js-lee-AI/LeVoC_BPE
can be built by considering morphological factors and selecting only frequently-used subwords from the corresponding corpus. However, even with a subword that covers the entire vocabulary, the original meaning is not always preserved when segmented. This is due to the over-segmentation problem, in which the word’s meaning is distorted, as previously mentioned in Table 1.
Our study reveals that subword tokenization methods such as BPE, which construct vocabularies in a language-agnostic manner, are particularly susceptible to over-segmentation issues. We attribute this to the fact that the existing subword tokenization methods generally rely on the statistical analysis of occurrence frequencies and do not consider long words2. In the case of BPE or SentencePiece (Kudo and Richardson, 2018), increasing the vocabulary size does not solve the oversegmentation problem and leads to delayed parameter updates in the embedding layer (Cherry et al., 2018; Ding et al., 2019).
To address this limitation, we propose intentionally increasing the vocabulary composition ratio for long words. Specifically, we hypothesize that over-segmentation issues must be considered to achieve high translation performance in Korean after addressing the OOV problem by increasing the number of BPE merge operations.
Methodology We denote V{} as the vocabulary of the BPE model BPE{}. The LeVoC method is designed to augment the proportion of long tokens in the vocabulary. Rather than employing the BPE model derived from a parallel corpus P , LeVoC employs a BPELeVoC that exhibits a higher proportion of long tokens. The vocabulary VLeVoC is formulated by merging the vocabularies obtained from the BPE model and the long token set. When the target vocabulary size for BPELeVoC is s, we train the BPEsmall with a vocabulary size of s(1 − r), where r is a hyperparameter set between 0 and 1 to denote the proportion of added long tokens in VLeVoC. The pseudocode for constructing LeVoC is illustrated in Figure 1.
Step 1. Employ a large monolingual Korean corpus, M , capable of extracting long tokens. Then divide M into two corpora: MExt for extracting long tokens and MSel for selecting tokens to be included in VLeVoC.
Step 2. By utilizing MExt, train BPEExt with a 2In this study, we define “long word” or “long token” as a
word which character length is longer than 4.
vocabulary size of sExt and BPEsmall with a vocabulary size of s(1− r).
Step 3. Among the subwords included in VExt, extract lengthy tokens which character length is longer than L to create a bag of subwords, BOWExt. L is priorly set by a hyperparameter.
Step 4, 5. VLeVoC is formulated by adding frequently occurring long tokens from BOWExt to Vsmall. Specifically, we add tokens in BOWExt sequentially until the number of added tokens reaches s ∗ r. We prioritize subwords with high token frequency in MSel but exclude subwords not present in P .
Through these procedures, we can formulate VLeVoC with a deliberately increased distribution for long tokens.

heading: 3.2 LeVoC Hyperparameter Setting
text: We put forth a series of recommendations for hyperparameter configurations. (i) When constructing LeVoC for generic domain NMT models, it is imperative to ensure that the number of words included in the context set M is sufficiently large. This is because, during the NMT training process, the model’s performance may decline if the long tokens added from BOWExt are not included or are under-represented in the NMT training corpus.
This can result in less updated word embeddings in the corresponding embedding layer (Luong et al.; Schick and Schütze, 2020; Yu et al., 2022). We suggest setting the context set to contain more than 0.2 million words, and in our experiments, we utilize a context set of approximately 1 billion words to increase diversity by including a wide range of words. (ii) We recommend setting the value of the sExt to be sufficiently large to ensure that various long tokens are covered. We found that increasing the value of sExt allows for extracting various tokens that maintain the morphological structure in our pilot study. In our experiments, we set the value of sExt to 1024k. (iii) The minimum length of long tokens, L, is recommended to be set to 4 or 5 for the Korean language. Table 2 shows that in vocabularies with fewer than 64k tokens, up to 98% of token lengths are less than 4. To capture a diverse range of long tokens that preserve the morphological structure, we set L to 4 in our primary experiments, and 5 in our case study (in Appendix A). (iv) We found that either a value of r of 0.2 or 0.4 effectively maintains the performance of the original BPE while adequately incorporating long tokens. In all our experiments, we selected the higher performing value of r between these two.

heading: 4 Experimental Setup

heading: 4.1 Dataset Details
text: We employed the AI-Hub3 Korean-English parallel corpus to assess machine translation performance. The AI-Hub dataset, comprising approximately 1.6 million sentences, ensures high quality through human verification. We allocated substantial weight to the validation and test sets. Specifically, we randomly selected approximately 1.3 million samples from the AI-Hub data for the training set, 250,000 samples for the validation set, and the remaining for the test set. Detailed information on the dataset is provided in Appendix B.
3https://aihub.or.kr/
Moreover, our experimental results on vocabulary corpus selection indicated that in the baseline BPE, a larger corpus (18 GiB) delivers superior performance compared to the only use of the AI-Hub corpus. As a result, we integrated a larger corpus into all experiments, including the baselines, to ensure a consistent and thorough evaluation. Additional information on corpus selection can be found in Appendix C.

heading: 4.2 Experimental Design
text: We utilized the BPE method as a baseline in our machine translation and morphological coverage experiments. We evaluated the performance of the models trained with the combination of English vocabulary of size 16k and Korean vocabulary of sizes 16k, 32k, and 64k.
In assessing machine translation performance, we train a Transformer (Vaswani et al., 2017) model from scratch and compared the translation performance of our proposed LeVoC-BPE with that of BPE. We also trained all machine translation models using Fairseq (Ott et al., 2019) with identical hyperparameters. The specific values of these hyperparameters are provided in Appendix D.
To evaluate morphological coverage, we extract morphologically rich words of four or more characters from both out-of-domain and in-domain corpora using the morpheme segmenter mecab-ko4. These words were subsequently used as an evaluation set for morpheme segmentation.

heading: 4.3 Evaluation Details
text: We assessed the performance of each NMT model using the SacreBLEU score (Papineni et al., 2002; Post, 2018), and chrF++ (Popović, 2015) metrics, with a beam size of 5 for all cases. The models were evaluated on the FLoRes dev-test sets (NLLB Team, 2022)5 for out-of-domain comparison, and the divided AI-Hub test sets, for indomain comparison. To evaluate Korean sentences, we segmented them into morpheme units, as their performance can be unduly underestimated depending on postpositions, suffixes, and prefixes (Park et al., 2020; Bandyopadhyay et al., 2021). For this purpose, we employed the segmenter, mecab-ko.
To explore the correlation between enhancements in machine translation performance and the morphological quality of long tokens, we report the
4https://bitbucket.org/eunjeon/mecab-ko-dic/ 5We utilized this dataset under a CC-BY-SA 4.0 license.
average subword count and utilize full-match (Hofmann et al., 2022). The average subword count, indicative of a token’s morphological information, is computed as the mean number of subwords encompassing all morpheme elements with 100% coverage. On the other hand, full-match denotes the accuracy of subwords that preserve morphological words intact, thereby facilitating the verification of tokenization accuracy. According to the definition of full-match by Hofmann et al. (2022), it occurs when the tokenization precisely aligns with the gold segmentation. This measurement is particularly suitable for assessing the mitigation of oversegmentation of long words, thereby providing a robust framework for evaluating our hypothesis regarding the performance.
The results regarding the precision, recall, and F1 scores for the morpheme boundary of tokenizers can be found in Appendix E. Briefly, LeVoC exhibits a higher recall and F1-score compared to BPE when given the same vocabulary size, indicating its superior performance in detecting reference boundaries.

heading: 5 Experimental Results
text: LeVoC with BPE Table 3 presents the evaluation results of LeVoC-BPE in comparison with the BPE. The experimental results are consistent with the findings of Ding et al. (2019), who observed that smaller vocabulary sizes lead to improved performance. The proposed LeVoC models demonstrate superior performance to all the baselines in both Ko2En and En2Ko tasks. Notably, in the case of En2Ko, LeVoC exhibited an improvement of up to 2.13 BLEU and 1.75 chrF++ on the out-of-domain test set compared to the baselines. Additionally, LeVoC achieves an improvement of up to 0.84 BLEU
and 0.29 chrF++ on the in-domain test set. LeVoC-BPE outperforms in the target language. We conjecture that this is caused by some long tokens, which are seldom observed in the machine translation training dataset, resulting in inadequately trained embeddings for these tokens (Schick and Schütze, 2020). The Transformer (Vaswani et al., 2017) architecture can effectively decode context-appropriate tokens by suppressing the generation of infrequently trained long Korean tokens in En2Ko. However, in Ko2En, where rare tokens are invariably encoded with the corresponding embedding vector, parameters for the infrequent words may not be adequately trained. This can cause the corresponding parameters to resemble a random embedding and fail to accurately reflect contextual information.
Moreover, we observe that LeVoC demonstrates superior translation performance on both out-ofdomain and in-domain test sets. While in-domain performance is less affected by over-segmentation and the resulting semantic confusion, we observe that the impact becomes significantly more pronounced in an out-of-domain context. This finding aligns with our motivation, which emphasizes that over-segmentation during the training phase can systematically degrade the performance of translation models, particularly compromising reliability when faced with complex or out-of-domain texts. Given that LeVoCs are trained using identical hyperparameters, vocabulary size, and dataset as BPE, the results imply that LeVoC can rectify the issue of over-segmentation in the Korean language and be generalized for other datasets. Furthermore, an analysis of LeVoC’s performance with downscaled training data sizes is presented in Section 6.3. In summary, it is observed that as the quantity of train-
ing data decreases, the performance gains provided by LeVoC diminish. This observation supports our initial premise that over-segmentation introduces accumulated noise into translation performance, which has a more pronounced effect in scenarios with larger training datasets.
The comparative analysis of the performances, using only the training corpus without any external corpus, is presented in Appendix F. Succinctly, barring the 64k vocabulary size, the performance of LeVoC trained with an external corpus surpasses that of LeVoC trained only with the training corpus. In addition, a comparison between Unigram (Kudo, 2018) and LeVoC-Unigram, with further experimental results, is provided in Section 6.5.
Preservation of Morphological Information Table 4 presents the experimental results regarding the preservation of morphological information. For vocabularies of equivalent size, LeVoC-BPE exhibits a substantial improvement in full-match scores compared to the baseline BPE method. However, LeVoC-BPE displays a minor decrease in terms of average subword count performance.
Our findings indicate that LeVoC can augment the morphological quality compared to the BPE with an equivalent vocabulary size. A high fullmatch rate signifies the ability to encode morpho-
logical information directly via the added long tokens. Despite achieving superior full-match performance, LeVoC exhibits a slightly lower performance of average subword count than BPE. The results suggest that even LeVoC cannot incorporate all the long tokens encountered during the training or inference phase. Nevertheless, LeVoC can segment the corresponding tokens at a finer granularity without the OOV problem.
The distribution of the token lengths for BPE and LeVoC-BPE vocabularies are depicted in Figure 2. As illustrated, LeVoC exhibits a higher composition ratio in long tokens, while BPE displays a skewed distribution towards shorter tokens. These distributions may serve as a crucial factor in achieving higher morphological full-match rates.

heading: 6 Case Studies

heading: 6.1 Impact on Morphologically Rich Setup
text: We propose that the successful translation of long words contributes significantly to the enhanced performance of LeVoC, primarily due to the distribution of token length within the same vocabulary size. To validate this, we present two scenarios: (1) Evaluation of the morphologically rich setting (MORPH-RICH), where all sentences contain a minimum of five words, each with at least four char-
acters, and (2) Evaluation of the morphologically ideal setting (MORPH-IDEAL), where each sentence includes at least five long words, each with a minimum of four characters that are found within the LeVoC vocabulary. The experimental results are displayed in Table 5. It is crucial to note that while preserving morphological information is a significant factor in LeVoC’s high MT performance, other factors may also contribute to preserving morphological information.
Our findings indicate that LeVoC surpasses BPE in both settings, particularly demonstrating superior performance6 in the MORPH-IDEAL setting. We deduce that the preservation of morphological information contributes to the high machine translation performance of LeVoC in these settings. In our prior experiments in Section 5, we discovered that LeVoC-BPE exhibits higher morphological full-match performance than BPE, suggesting that LeVoC-BPE can preserve morphological information within each sentence more effectively than BPE. This advantage may significantly enhance the performance of LeVoC in the MORPH-RICH and MORPH-IDEAL settings, as evidenced by the wider performance gap between LeVoc-BPE and BPE in these settings.

heading: 6.2 Morpheme-aware Supervised Method Setup
text: Using a morpheme segmenter is one of the key solutions for enhancing morpheme coverage. Morpheme segmenters are perceived as effective tools for integrating linguistic information. In practical terms, incorporating a morpheme segmenter into a BPE tokenizer can significantly improve Korean MT systems and is currently considered the
6A direct comparison between LeVoC-BPE and MorphBPE may not yield accurate results. LeVoC-BPE does not perform BPE at the morpheme level, unlike Morph-BPE which employs the same morpheme splitter, mecab-ko, for BPE at the morpheme level. Comparing these two methods could lead to an imbalanced and potentially unfair setup.
state-of-the-art tokenizer (Park et al., 2020, 2021). In this section, we apply LeVoC to a morphemebased BPE tokenizer (Morphs-BPE) and investigate whether we can achieve additional performance improvement. The Morphs-BPE model used in our experiments is a Korean-specific state-ofthe-art tokenizer that applies morpheme-based segmentation followed by BPE training (Park et al., 2020). We apply our proposed method, LeVoC, to the Morphs-BPE tokenizer, resulting in the LeVoCMorphs-BPE. The translation performances of this tokenizer are outlined in Table 6.
Experimental results indicate that LeVoC improves all BLEU and full-match scores of MorphsBPE cases, with a maximum enhancement of 0.56 in BLEU score and 11.75 in full-match performance. This demonstrates the high applicability of LeVoC and suggests that intentionally increasing the composition ratio of long tokens in a BPE vocabulary provides a significant advantage in Korean MT models. Furthermore, these results confirm that even a morpheme-aware BPE utilizing a morpheme analyzer can still face the issue of over-segmentation. The token length distributions of both tokenizers are described in Appendix G.

heading: 6.3 Impact of Training Data Size
text: Table 7 shows the impact of training data size on the performance of the LeVoC method. It was ob-
served that as the volume of training data decreases, the performance gains afforded by LeVoC diminish. This trend may be attributed to two primary factors: (i) there may not be enough training corpus for the longer tokens added to LeVoC to be trained with appropriate embedding values, and (ii) a reduction in semantic confusion due to over-segmentation as the quantity of training data is scaled down. These observations lend support to our initial premise that over-segmentation exerts a more substantial influence on translation performance in scenarios where the training dataset is larger. The results indicate that while LeVoC can improve translation performance, its benefits are more pronounced with larger amounts of training data, where over-segmentation is more likely to cause semantic confusion.

heading: 6.4 LeVoC to Another Language
text: To investigate the impact of another language on LeVoC, we extended our experiments to include the German language, known for its morphological complexity. Utilizing a subset of the WMT19 EnDe News Translation Task dataset7, we maintained a consistent data setup with 1.3 million training pairs and 250,000 validation pairs, mirroring the Korean data configuration.
While the application of LeVoC to Korean showed consistent performance improvements over BPE across both vocabulary sizes and domains, the same approach applied to German, as shown in Table 8, yielded only marginal improvements. This suggests that the effectiveness of the LeVoC method may vary depending on the language.

heading: 6.5 LeVoC in Unigram Setup
text: Table 9 presents the outcomes of our initial examination on the performance comparison between Unigram (Kudo, 2018) and LeVoC-Unigram in English to Korean translation. The table elucidates the enhancements achieved by LeVoC when applied to both BPE and Unigram tokenization methods. It is observed that LeVoC consistently improves
7http://www.statmt.org/wmt19/
performance in out-of-domain settings for both tokenization methods, albeit with a smaller margin of improvement compared to BPE. In contrast, in in-domain settings, LeVoC-Unigram exhibits lower performance than Unigram alone.

heading: 7 Qualitative Analysis
text: Segmentation and Translation Fidelity In Table 10, we provide an empirical case demonstrating the adverse effects of BPE over-segmentation on the Korean-to-English translation task and how LeVoC ameliorates these issues. The BPE method dismantles the word “상대적으로 (relatively)” into subwords that lose their meaning, while LeVoC preserves the morphological integrity of the original word, resulting in a more accurate translation. This instance highlights the necessity of incorporating morphological considerations during segmentation to retain the intended meaning and enhance translation quality.
However, the implications of over-segmentation are not limited to isolated translation errors; it represents a systemic issue within NMT that introduces semantic noise across the training dataset. This noise impairs the model’s ability to learn and generalize language patterns, leading to a potential decline in performance, especially with complex or non-standard texts. Initial training stages may exhibit frequent word-level mistranslations due to over-segmentation, but these errors become less common as training progresses. Nonetheless, persistent semantic noise continues to affect the model’s performance adversely.
LeVoC addresses this issue by prioritizing including longer, morphologically rich subwords into the vocabulary, which supports the learning process by providing a clearer semantic signal.
Morphological Structure Preservation As depicted in Table 11, the results of BPE and LeVoC encoding examples for the in-domain test set are presented. Rows 1-3, 4-6, and 7-9 of the table correspond to cases 1, 2, and 3 outlined in Table 1, respectively. In the first case (rows 1-3), BPE decomposes the word “직장에서 (In workplace)” into “직 (Null meaning)” + “장에서 (In the place)”, resulting in the loss of the word’s meaning. In the second case (rows 4-6), BPE segments “알아보 는 (Recognizing)” into two other words with almost irrelevant meanings as “알아 (Know)” + “보 는 (Looking).” Similarly, in the last case (rows 7-9), “의사결정 (Decision-making)” is segmented into “의사 (Doctor, Opinion)”, which has ambiguous meanings, and “결정 (Decision).” Conversely, LeVoC includes these words in its vocabulary and encodes them as “직장에서 (In workplace)”, “알 아보는 (Recognizing)”, and “의사결정 (Decisionmaking)”, without disassembling the words. Therefore, LeVoC can maintain a more accurate morphological structure compared to BPE. In Appendix H, we further present a qualitative analysis of extensive examples of generative results in English-toKorean translation.

heading: 8 Conclusion
text: In this paper, we focus on the over-segmentation issue in the Korean MT models that erodes morphological information in the text. We discovered that conventional BPE tokenizers, while effectively alleviating the OOV problem, are prone to oversegmentation. We found that this semantic confusion, which stems from over-segmentation, ultimately degrades the overall translation quality. To address this issue, we propose LeVoC, a simple and effective method for mitigating over-segmentation by preserving the morphological structure of long tokens. We found that LeVoC can notably enhance the MT performance and preserve morphological information, especially for morphologically rich and ideal settings. LeVoC can be seamlessly integrated with a morpheme analyzer designed for the Korean language to maintain morphological integrity. Our comprehensive evaluation revealed that addressing over-segmentation via manipulating token length distribution in a BPE vocabulary improves MT performance substantially. We hope that the findings from our study will inspire further research in natural language processing by considering Korean’s unique characteristics.
Limitations
The LeVoC methodology, while promising, has certain limitations that should be acknowledged.
Focus on Korean Over-Segmentation The focus of this study was specifically on addressing the over-segmentation issue in Korean language, which is particularly pronounced when using the BPE due to the unique morphological richness of the language. This focus on BPE-driven oversegmentation in Korean is the primary reason why our experiments were not extended to other subword methods such as Unigram (Kudo, 2018) or Wordpiece (Schuster and Nakajima, 2012).
Language-Dependent Efficacy Our generalization study, as shown in Section 6.4, highlighted another significant limitation of the LeVoC methodology. While applying LeVoC to Korean consistently yielded performance improvements over BPE across different vocabulary sizes and domains, the same method when applied to German only resulted in marginal improvements. This limitation was also observed in our additional generalization experiments with other languages such as Mongolian and Japanese, which are not included in our experiments. These findings suggest that the efficacy of the LeVoC method may be languagedependent and potentially influenced by specific linguistic characteristics, such as the setting of the hyperparameter L. The discrepancy in performance improvement underscores the importance of considering the unique linguistic characteristics of each language when developing tokenization strategies. Further research is needed to understand how LeVoC can be effectively adapted to other morphologically rich languages.
Vocabulary Size Variation Due to computational resource constraints, we were unable to conduct experiments across a diverse range of vocabulary sizes. The impact of adding morphological words to the vocabulary with varying amounts of external data and different vocabulary sizes remains unexplored.
Domain-Specific Experiments Lastly, our experiments were primarily confined to the domain of neural machine translation. The applicability and effectiveness of the LeVoC methodology in other areas such as language modeling and languageagnostic segmentation have not been investigated and remain potential avenues for future research.
Ethics Statement
In our study, we utilized datasets in which potentially identifiable sentences present in the wikitext8 had been removed by participants. Additionally, we utilized the official datasets that effectively mitigated concerns regarding aggregation privacy, thus ensuring that the data did not contain any personally identifiable information. Consequently, our work does not incorporate any potentially harmful sentences. However, it is crucial to acknowledge that, as with any general translation system, potential biases, such as those related to gender, may still be present (Stanovsky et al., 2019).

heading: Acknowledgments
text: This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIT) (RS-2024-00398115, Research on the reliability and coherence of outcomes produced by Generative AI). This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(NRF2021R1A6A1A03045425). This work was supported by Institute for Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2022-0-00369, (Part 4) Development of AI Technology to support Expert Decision-making that can Explain the Reasons/Grounds for Judgment Results based on Expert Knowledge).

heading: A Observations on Minimum Subword Length L
text: In this section, we experiment on the minimum subword length L that determines the standard for the “long token.” We perform the experiments on the out-of-domain test set to inspect more general standards, and the results are presented in Table 12.
In the English-to-Korean translation task, we found that a minimum subword length of L = 4 outperforms L = 5 in all the cases. While L = 5 enables the extraction of more morphologically rich subwords, it limits the inclusion of words of length 4 in the vocabulary, resulting in a lower full-match rate. Additionally, using L = 5 necessitates a larger corpus to maintain the desired vocabulary size. Based on these results, we conclude that a minimum subword length of L = 4 is appropriate for use in LeVoC.

heading: B Dataset
text: BPE Corpus The corpus employed in our experiments, referred to as the BPE corpus, comprises the following sources: Wikitext9 (with potentially identifying sentences removed by study participants), ModuCorpus10 (including messenger, written, colloquial, and web data), AI-Hub news data, AI-Hub MT corpus11 (repeated five times), and a validation corpus. The total raw data amounts to 18.5 GB. By removing date or time notations (e.g., (20191115 09:48)) and repeated words (e.g., blah blah, ....), we obtained a corpus of 17.9 GB. The statistics of this corpus used to construct the vocabularies are presented in Table 13.
AI-Hub The AI-Hub dataset comprises a total of 1.6 million sentences across five domains: news (0.8 million), websites (0.1 million), culture (0.1 million), ordinance (0.1 million), and spoken language (0.5 million). This is a Ko↔En parallel dataset, with an error rate of less than 1%. We divide the samples into train, validation, and test sets to analyze the experimental results while maintaining the domain ratios. The number of sentence pairs in the separated samples is presented in Table 14.
9https://dumps.wikimedia.org/kowiki/. We utilized this dataset under a CC BY-SA 3.0 license. 10https://corpus.korean.go.kr/ 11https://www.aihub.or.kr/

heading: C Selection of Vocabulary Corpus
text: Table 15 shows the result of our pilot study that trains the transformer-based Ko2En NMT model using the vocabulary constructed through the large corpus (18GB) and AI-Hub. All Vsmall, VExt, and baseline BPE are trained through a large corpus for a fair highest performance comparison.

heading: D MT setting details
text: We adopt the fairseq framework (Ott et al., 2019) for training the NMT model. To ensure a fair comparison, we train a Transformer-base NMT architecture from scratch using the same hyperparameters12. We train a MT model for each setting and the corresponding training arguments are noted in Figure 3.
12We trained each model for approximately 30 hours with two NVIDIA RTX 6000 GPUs.

heading: E Morpheme Boundary Analysis with LeVoC and BPE
text: We utilized mecab-ko to generate reference boundaries from out-of-domain data, excluding all English words. In all cases, LeVoC exhibits lower precision but higher recall and F1-score compared to BPE when given the same vocabulary size. The diminished precision implies that LeVoC identifies boundaries even where actual morpheme boundaries do not exist, resulting in increased segmentation. The elevated recall signifies that LeVoC outperforms BPE in detecting reference boundaries, indicating that it does not overlook many genuine morpheme boundaries.

heading: F Evaluating LeVoC with Only Training Corpus
text: In certain situations, an additional corpus may not be available. In such cases, we evaluate the performance of the MT model by configuring LeVoC using only the training corpus, i.e., using P to build BPELeVoC. For sExt, we adopt 512k instead of 1024k due to the training corpus size. Under this condition, VExt can not be significantly increased unless the training corpus is extensive, meaning that the number of long tokens that can be extracted may be limited. The results are presented in Table 17.
The performance of the LeVoC model trained using only the training corpus is generally similar to that of the LeVoC model trained using the additional large corpus, with the latter exhibiting slightly better performance.

heading: G Token Length Distribution of Morpheme Analyzer-based Vocabulary
text: We intentionally upscale the distribution of long subwords (with more than four character lengths) when constructing the LeVoC-BPE and LeVoC-Morphs-BPE vocabularies. Figure 4 explains the token length distribution of the vocabulary’s lexical list generated by the morpheme analyzer-based approach.

heading: H Examples of English-to-Korean Translation
text: For more extensive qualitative analysis, we provide inference examples of our NMT models. We compare the Korean-to-English BPE model to LeVoC-BPE by sampling low BLEU examples in the out-of-domain test set. The compared results are shown in Table 18. LeVoC-BPE decodes long words into long tokens, while BPE segments them into tokens of relatively constant length.

heading: Future Work
text: The applicability and effectiveness of the LeVoC methodology in other areas such as language modeling and languageagnostic segmentation have not been investigated and remain potential avenues for future research