## 3 Word Segmentation and Vocabulary Construction  
In the field of natural language processing [16], text preprocessing is the foundation for building efficient models, and the word segmentation step is particularly important, as it directly affects the model's understanding and processing efficiency of the text [17]. Currently, some mainstream word segmentation tools include jieba, SentencePiece, and HanLP. As shown in Table 2, each word segmenter has its unique design and applicable scenarios:  
* Jieba, as a classic Chinese word segmentation library, is renowned for its maturity, stability, and rich customization capabilities. It is particularly adept at handling word segmentation in Chinese text, but it may have limitations in dealing with unknown words (new words or specialized terminology) and long sentence structures.
* HanLP, as a comprehensive natural language processing toolkit [18], offers a wide range of Chinese processing functions, including word segmentation, part-of-speech tagging [19], etc. Its advantages lie in its high configurability and deeply optimized models. However, for large-scale data processing and task-specific optimization needs, more customization work may be required.
* In contrast, SentencePiece stands out as our preferred word segmentation and vocabulary construction tool in this study due to its unique design philosophy and practical advantages:
* Flexibility and universality: SentencePiece is not limited to a specific language. Its statistical-based subword unit generation method enables it to excel in multi-language processing, especially for mixed-language or low-resource language text data.
* Adaptability: By learning statistical patterns in the data [20], SentencePiece can automatically generate a subword dictionary that best fits the current corpus. This gives it significant advantages in discovering new words and handling rare words, which is particularly important for rapidly changing internet language and specialized literature.
* Efficiency and practicability: It supports multiple model types and adopts the unigram model by default, balancing vocabulary coverage and model complexity. At the same time, the generated integer encoding effectively reduces storage space requirements and accelerates subsequent model training processes.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Tokenizer Tool** & **Advantages** & **Disadvantages** & **Uyghur** & **Chinese** & **English** & **Japanese** \\  & & & **Language** & **Language** & **Language** & **Language** \\ \hline \hline \multirow{6}{*}{jieba} & 1. Supports three tokenization & \multirow{2}{*}{1. Ineffective recognition of proper nouns and rare words engine mode [22]} & \multirow{2}{*}{\(\times\)} & \multirow{2}{*}{\(\surd\)} & \multirow{2}{*}{\(\surd\)} & \multirow{2}{*}{\(\surd\)} \\  & & & & & & \\ \cline{1-1}  & & & & & & \\ \cline{1-1}  & & & & & & \\ \cline{1-1}  & & & & & & \\ \cline{1-1}  & & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Function Characteristics Table of Mainstream Word Segmenters [21].  
After comparing various tokenizers such as jieba, SentencePiece, and HanLP, we have ultimately decided to adopt SentencePiece as the core tokenization technology for this research. It not only satisfies the need for efficiently processing large-scale text data, but also provides a solid foundation for subsequent natural language model training through its flexible subword strategy, particularly demonstrating unique advantages in balancing model performance and generalization capabilities. Tokenization is a crucial step in the preprocessing phase, aiming to effectively convert raw text into serialized numerical representations that the model can understand. To achieve this goal, we have employed the SentencePiece model, a highly efficient subword tokenization tool widely used in the field of natural language processing [23].