Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languages
Authors: Wenhao Zhuang, Yuan Sun, Xiaobing Zhao
Accepted at: acl25

Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languages

Wenhao Zhuang1,2, Yuan Sun1,2,3,* , Xiaobing Zhao1,2,3
1Minzu University of China, Beijing, China
2National Language Resource Monitoring & Research Center Minority Languages Branch
3Institute of National Security, Minzu University of China, Beijing, China

Emails: sdrz_zwh@163.com, sunyuan@muc.edu.cn, nmzxb_cn@163.com
* Corresponding author: Yuan Sun

Abstract
As large language models (LLMs) are trained on increasingly diverse and extensive multilingual corpora, they demonstrate cross-lingual transfer capabilities. However, these capabilities often fail to effectively extend to low-resource languages, particularly those utilizing non-Latin scripts. While transliterating low-resource languages into Latin script presents a natural solution, there currently lacks a comprehensive framework for integrating transliteration into LLMs training and deployment. Taking a pragmatic approach, this paper innovatively combines character transliteration with Huffman coding to design a complete transliteration framework. Our proposed framework offers the following advantages: 1) Compression: Reduces storage requirements for low-resource language content, achieving up to 50% reduction in file size and 50-80% reduction in token count. 2) Accuracy: Guarantees 100% lossless conversion from transliterated text back to the source language. 3) Efficiency: Eliminates the need for vocabulary expansion for low-resource languages, improving training and inference efficiency. 4) Scalability: The framework can be extended to other low-resource languages. We validate the effectiveness of our framework across multiple downstream tasks, including text classification, machine reading comprehension, and machine translation. Experimental results demonstrate that our method significantly enhances the model’s capability to process low-resource languages while maintaining performance on high-resource languages. Our data and code are publicly available at https://github.com/CMLI-NLP/HuffmanTranslit.

1 Introduction
Large language models have demonstrated remarkable multilingual transfer capabilities, enabling knowledge transfer from one language to another without additional training (Qi et al., 2023; Gao et al., 2024; Ye et al., 2023). However, this transfer ability often performs poorly in low-resource languages, primarily constrained by three factors: scarcity of training data (Costa-jussà et al., 2022), insufficient cross-lingual word embedding alignment (Deshpande et al., 2021), and writing system differences (Anastasopoulos and Neubig, 2019; Muller et al., 2021).

Common approaches to improving LLMs’ adaptability to low-resource languages include continued pre-training and supervised fine-tuning (Tao et al., 2024). Due to the low representation of low-resource languages in tokenizers and frequent occurrence of UNKnown tokens (Moosa et al., 2023), vocabulary expansion becomes a primary task (Zhuang and Sun, 2025). However, ensuring high performance for multiple low-resource languages is extremely challenging, facing two key issues. The first issue is the increased training and inference costs due to vocabulary size, as vocabulary must inevitably expand with the addition of languages to ensure tokenization performance for each language (Purkayastha et al., 2023). The second issue is the curse of multilinguality, which means that using a fixed-capacity model to pre-train multiple languages can improve cross-lingual performance to some extent, but performance begins to decline beyond a certain point (Conneau, 2019).

To improve cross-lingual transfer while avoiding issues associated with extensive vocabulary expansion, transliteration of low-resource languages has emerged as a viable approach. Transliteration refers to the process of converting text from one writing system to another according to specific rules, typically converting non-Latin scripts to Latin alphabet representation (Wellisch, 1978). Previous studies have demonstrated that transliterating text into a common character set can enhance cross-lingual transfer performance for low-resource languages with non-Latin scripts (Liu et al., 2024a, 2025). This improvement is attributed to the common character set facilitating knowledge transfer
through lexical overlap (Dhamecha et al., 2021; Pires, 2019; Amrhein and Sennrich, 2020) and enabling the reuse of existing information in embedding matrices (Purkayastha et al., 2023). Appending transliterated content to prompt templates for low-resource languages has been shown to improve downstream task performance (Ma et al., 2024).

Most research on transliteration relies on existing tools like UROMAN (Hermjakob et al., 2018), which maps any UTF-8 character to Latin letters, to investigate the impact of transliteration on cross-lingual transfer or alignment. However, this transliteration process is irreversible; due to the potential loss of characteristic information from the original script during transliteration, it is impossible to accurately restore the transliterated Latin letters back to the source language script, which limits its practical applications (Amrhein and Sennrich, 2020). Furthermore, low-resource languages typically utilize extended Unicode character sets for encoding, resulting in their textual data occupying more storage space compared to languages like English. This storage overhead issue becomes more prominent when processing large-scale multilingual corpora. Therefore, this paper focuses on two key issues: how to implement a reversible transliteration mechanism to facilitate practical applications while maintaining cross-lingual transfer effectiveness, and how to achieve text compression during the transliteration process to facilitate storage and training. We observe that these two points correspond precisely to the reversibility and compression properties of Huffman coding, which provides the theoretical foundation for our Huffman coding-based transliteration scheme.

We selected three low-resource languages: Tibetan, Mongolian, and Uyghur, which are minority languages in China with a total user base exceeding 30 million speakers, along with English and Chinese as high-resource languages for our experiments. We conducted continued pre-training of open-source LLMs using corpora obtained through various transliteration methods, analyzed cross-lingual transfer performance across text classification, named entity recognition, machine reading comprehension, knowledge extraction, and machine translation tasks, while also comparing compression rates among different transliteration methods. To enable the model to directly serve low-resource language users, we developed a FastText-based automatic transliteration framework that performs language detection before and after model processing, implementing transliteration and restoration of input and output, thereby maintaining native language interaction at the user end. In summary, our contributions are as follows:

• We propose a Huffman coding-based transliteration scheme for low-resource languages, achieving reversibility in the transliteration process and addressing the limitations of traditional transliteration methods in practical applications.

• Leveraging the compression properties of Huffman coding, we effectively reduce the storage overhead of low-resource language texts, making the training of large-scale multilingual corpora more efficient.

• We develop an end-to-end framework integrating FastText language identification, enabling automatic transliteration and restoration of low-resource languages while maintaining native language interaction and improving performance across multiple downstream tasks.

2 Related Works

Cross-lingual Transfer for Low-resource Languages To enhance low-resource language performance in LLMs, one primary approach uses continued pre-training and supervised fine-tuning with low-resource corpora (Wenhao et al., 2024). However, this method faces significant challenges: it requires complex vocabulary expansion and model architecture modifications, resulting in poor scalability, and most critically, suffers from limited training data availability (Joshi et al., 2020).

Alternative approaches focus on improving cross-lingual transfer capabilities through various mechanisms: concatenating multilingual input sequences to leverage shared representation spaces (Kim et al.; Tanwar et al., 2023; Cueva et al., 2024), projecting target language representations onto high-resource languages for enhanced feature extraction (Xu et al., 2023), and increasing the parallel content in multilingual training corpora (Zhuang and Sun, 2025).

While these methods aim to transfer capabilities from resource-rich to low-resource languages, a fundamental challenge remains: the substantial differences in writing systems among low-resource languages. Unifying multiple languages into a single writing system could potentially address vocabulary challenges and promote vocabulary sharing,
thereby facilitating cross-lingual knowledge transfer (Purkayastha et al., 2023).

Tokenization and Vocabulary Expansion Existing subword tokenizers (such as BPE (Sennrich, 2015) and SentencePiece (Kudo, 2018)) have been widely adopted for low-resource languages. However, due to the limited representation of these languages in pre-training corpora, they suffer from insufficient vocabulary coverage, over-segmentation, and high ratios of unknown tokens. While vocabulary expansion (Cui et al., 2023) offers a potential solution, it introduces new challenges: the need for substantial training data to adequately train new tokens, and increased model capacity requirements to mitigate the multilingual curse (Conneau, 2019).

Recent approaches have focused on more efficient solutions, such as leveraging shared linguistic information and cross-lingual word embedding alignment (Ogueji et al., 2021; Liu et al., 2021), which improve tokenization without significant vocabulary expansion. Notably, transliterating low-resource languages into a unified writing system has shown promising results (Dhamecha et al., 2021; Liu et al., 2024b), simultaneously enhancing vocabulary sharing and model transfer capabilities while avoiding the computational overhead of vocabulary expansion.

Romanization and Transliteration Romanization is the process of mapping various characters to Latin characters, though this process is typically irreversible. Its objective is to approximate the pronunciation of the original character text as closely as possible. Specialized tools like UROMAN (Hermjakob et al., 2018) can romanize almost all characters by directly mapping UTF-8 characters to Latin letters, though this process involves information loss, such as the omission of tonal information. There are also general character conversion tools like uconv that can preserve more original character information, such as adding diacritical marks, but this limits subword sharing across languages.

The Tibetan, Mongolian, Uyghur, and Chinese languages used in our experiments can all be romanized through UROMAN; however, due to the uniqueness of the Tibetan writing system, uconv currently cannot transliterate Tibetan. Romanization encoding has been studied in both natural language processing and speech processing domains, such as its application in multilingual pre-trained language models to enhance low-resource languages (Purkayastha et al., 2023), and in speech processing systems’ pre-training as additional forced alignment for text labeling (Pratap et al., 2024). Moreover, phonological distinctions may be lost during romanization - for instance, Chinese characters become toneless pinyin when romanized, with a single pinyin potentially corresponding to many different characters. UROMAN also converts numbers from different writing systems into Western Arabic numerals (Ding et al., 2024), which further complicates the process of converting romanized text back to source languages, particularly when users expect LLMs to output in their native writing systems. In contrast, our proposed Huffman coding-based transliteration method is an innovative approach that balances transliteration (improving cross-lingual transfer), compression (reducing storage and training costs), and reversibility (facilitating practical restoration and interaction).

3 Methodology

3.1 Overview

We propose a three-stage processing approach for low-resource language transliteration and applications: (1) character encoding design: analyzing character frequencies and designing custom encodings. (2) transliteration and model training: training on transliterated raw corpora. (3) end-to-end language processing pipeline: comprising input language classification and processing, model inference, and output language classification and processing, as shown in Figure 1.

In this study, we focus on three low-resource languages of China: Tibetan, Uyghur, and Mongolian (see Table 1). These languages represent different writing systems, with a combined user base exceeding 30 million speakers. We select these languages because they present significant challenges for cross-lingual transfer: they use non-Latin scripts, have limited digital resources, and exhibit distinct writing systems that differ substantially from high-resource languages like Chinese and English.

<table>
  <tr>
    <th>Name</th>
    <th>ISO 639-1</th>
    <th>Writing System</th>
  </tr>
  <tr>
    <td>Tibetan</td>
    <td>bo</td>
    <td>Tibetan script</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>ug</td>
    <td>Uyghur Arabic script</td>
  </tr>
  <tr>
    <td>Mongolian</td>
    <td>mn</td>
    <td>Traditional Mongolian script</td>
  </tr>
</table>

Table 1: Overview of the low-resource languages studied in this work.
Step 1: Character Encoding Design

Step 2: Transliteration and Model Training

Step 3: Language Processing Pipeline

Figure 1: Overview of our three-stage approach. Step 1: Character encoding design with Unicode character statistics and custom encoding assignment. Step 2: Transliteration process for model training input. Step 3: Language processing pipeline with language classification for user interaction.

3.2 Character Frequency Analysis

Since English already uses Latin script and Chinese has well-established romanization tools for converting characters to pinyin, we focused our transliteration efforts on three low-resource languages: Uyghur, Tibetan, and Mongolian. Our character frequency analysis began with the CUTE open-source parallel dataset (Zhuang and Sun, 2025), which provided aligned text across Chinese, Uyghur, Tibetan, and English. To incorporate Mongolian, which wasn’t originally included in CUTE, we followed the dataset’s methodology to collect and evaluate Mongolian translations, using human evaluators to assess translation quality from Chinese to Mongolian. For comprehensive character analysis, we sampled 3,000 instances from each of the three low-resource languages and conducted a thorough examination of their Unicode characters and frequencies. Through a systematic approach combining Unicode code point ranges, character naming conventions, and expert linguistic validation, we identified the core character sets for each language: 45 characters for Mongolian, 38 for Uyghur, and 81 for Tibetan. These character sets were then consolidated and arranged in descending order of frequency, providing a foundation for our transliteration scheme.

3.3 Huffman-based Encoding Design

Properties of Huffman Coding  Huffman coding, as a variable-length encoding method, possesses both variable-length allocation and prefix properties. The variable-length allocation ensures that high-frequency characters receive shorter codes while low-frequency characters receive longer codes, providing a theoretical foundation for our text compression. The prefix property guarantees that no code is a prefix of any other character’s code, facilitating unambiguous decoding.

Customized Encoding Scheme  Based on the principles of Huffman coding, we designed an improved encoding scheme. To accommodate more languages, our scheme, while not strictly adhering to the prefix property, ensures unambiguous decoding through structured design. Specifically, we constrain all codes to follow the pattern "First letter capitalized, subsequent letters lowercase" and employ a maximum matching strategy for decoding.
Table 2 demonstrates the possibilities of this design pattern at different lengths.

<table>
  <tr>
    <th>Length</th>
    <th>Pattern</th>
    <th>Capacity</th>
  </tr>
  <tr>
    <td>One</td>
    <td>A, B, ..., Z</td>
    <td>26</td>
  </tr>
  <tr>
    <td>Two</td>
    <td>Aa, Ab, ..., Zz</td>
    <td>676</td>
  </tr>
  <tr>
    <td>Three</td>
    <td>Aaa, Aab, ..., Zzz</td>
    <td>17,576</td>
  </tr>
  <tr>
    <td>Four</td>
    <td>Aaaa, Aaab, ..., Zzzz</td>
    <td>456,976</td>
  </tr>
  <tr>
    <td colspan="2">Total (up to four characters)</td>
    <td>475,254</td>
  </tr>
</table>

Table 2: Encoding patterns and theoretical capacity for different lengths. The pattern consists of one uppercase letter followed by zero or more lowercase letters.

This design offers three key advantages: (1) By constraining the first character to be uppercase and subsequent characters to be lowercase, combined with the maximum matching strategy, it ensures unambiguous decoding. (2) It maintains the core principle of Huffman coding, allowing for variable-length code allocation based on character frequencies. (3) It provides significant scalability, theoretically supporting encoding for up to 475,254 characters. In this study, we implemented a subset of the two-character scheme, utilizing 21 single-character codes (B-X) and 141 two-character codes (Aa-Fk), totaling 162 encoding options. This scale is sufficient to cover the character sets of Uyghur, Tibetan, and Mongolian languages.

3.4 Transliteration Strategies

To explore optimal transliteration strategies, we designed three progressive transliteration schemes, with each scheme building upon and improving its predecessor.

Basic Transliteration Strategy We designed a transliteration scheme using "First-letter-capitalized + lowercase" encoding rules. This scheme transliterates Uyghur, Tibetan, and Mongolian into Latin alphabet representations according to encoding rules, while converting Chinese characters into Pinyin and preserving English text unchanged. While this strategy achieved basic transliteration functionality and reversibility, it did not account for the tokenizer’s characteristics, leaving room for further token compression.

Tokenizer-based Optimization Strategy To address the token optimization potential in the basic strategy, we analyzed the characteristics of the Llama2 tokenizer. Research showed that 66% of original characters required four tokens for representation. In response, we innovatively utilized single tokens from the Llama2 tokenizer (Touvron et al., 2023) as encoding mappings for original characters, enabling all 162 original characters to be represented by single tokens. The comparative token distribution is shown in Table 3.

<table>
  <tr>
    <th>Method</th>
    <th>1-token</th>
    <th>2-token</th>
    <th>3-token</th>
    <th>4-token</th>
  </tr>
  <tr>
    <td>Original</td>
    <td>1</td>
    <td>45</td>
    <td>9</td>
    <td>107</td>
  </tr>
  <tr>
    <td>Basic</td>
    <td>90</td>
    <td>72</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Optimized</td>
    <td>162</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
</table>

Table 3: Character distribution by token length after Llama2 tokenization. The Optimized method achieves single-token encoding for all characters.

Hybrid Vocabulary Strategy Building upon the second strategy, we leveraged the linguistic patterns inherent in the transliterated text to train a specialized vocabulary of 4,000 tokens. This vocabulary was merged with Llama2’s original 32,000-token vocabulary to create a hybrid vocabulary of 33,738 tokens, maintaining efficient single-character encoding while capturing common character combinations. The comparison of the three strategies is presented in Table 4. For a comprehensive analysis of file size and token compression ratios across different languages and strategies, see Appendix B.

<table>
  <tr>
    <th>Strategy</th>
    <th>Vocab Size</th>
    <th>Compr.</th>
    <th>Cost</th>
  </tr>
  <tr>
    <td>Basic</td>
    <td>32,000</td>
    <td>1.63x</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>Tokenizer</td>
    <td>32,000</td>
    <td>2.35x</td>
    <td>Medium</td>
  </tr>
  <tr>
    <td>Hybrid</td>
    <td>33,738</td>
    <td>3.04x</td>
    <td>High</td>
  </tr>
</table>

Table 4: Comparison of different strategies. Compr. shows average token compression ratio across Tibetan, Mongolian and Uyghur languages.

3.5 Reversibility Mechanism

Our transliteration system achieves perfect reversibility through a carefully designed mapping mechanism, maintaining bidirectional mappings between original characters and Latin codes. For characters not in the mapping tables (e.g., emojis, rare characters, or special symbols), the system preserves them using '@' markers with proper escape sequences (e.g., '@@' for the '@' character itself), ensuring no information loss during transliteration. The detailed process is shown in Algorithm 1.
Algorithm 1 Reversible Transliteration System

Input: \( M_{c2l}, M_{l2c} \): Bidirectional mappings, text: Input text
Output: Transliterated or restored text

Function ToLatin(text):
    result \( \leftarrow \emptyset \)
    for each c in text do
        Append \( M_{c2l}[c] \) if exists, else preserve as @...@
    end for
    return joined result

Function FromLatin(latin_text):
    result \( \leftarrow [], i \leftarrow 0 \)
    while \( i < \text{length}(latin\_text) \) do
        Process @ markers or find longest matching code
        Advance \( i \) accordingly
    end while
    return joined result

The system employs a greedy matching strategy during restoration, where it attempts to match the longest possible Latin code sequence for mapped characters while correctly handling preserved sequences between '@' markers. This dual mechanism ensures 100% restoration accuracy by either mapping characters through the bidirectional tables or preserving them in their original form.

3.6 Auxiliary Models for Practical Deployment

To achieve end-to-end system deployment, we developed three auxiliary models. At the input stage, we trained a FastText-based classifier specifically for identifying Mongolian, Tibetan, Uyghur, Chinese, and other languages. At the output stage, we trained a FastText language classifier tailored to the characteristics of transliterated text to guide language restoration. Additionally, to accurately handle the conversion from Chinese pinyin to characters, we fine-tuned a specialized model based on Qwen2.5-0.5B (Yang et al., 2024). The detailed training processes and evaluation results of these models are presented in Appendix A.

4 Experiments and Analysis

To evaluate the effectiveness of different transliteration schemes, we conducted a series of experiments examining the cross-lingual transfer performance of models trained with various transliteration strategies. We specifically focused on whether the models could effectively transfer knowledge to low-resource languages (Tibetan, Mongolian, and Uyghur) while maintaining performance in high-resource languages (Chinese and English).

4.1 Experimental Setup

We adopt the following experimental procedure: First, we process the pre-training corpus using different transliteration methods, followed by continued pre-training of the model. The choice of continued pre-training over training from scratch is motivated by the common challenge of insufficient training data faced by low-resource languages, which makes it difficult to support a complete pre-training process. After pre-training, we perform supervised fine-tuning using downstream task data from high-resource languages, and then directly conduct zero-shot evaluation on low-resource languages to verify the model’s cross-lingual transfer capability.

For Tibetan, Uyghur, and Mongolian languages, we identified a limited number of available datasets. Our experiments encompassed three primary tasks: text classification, machine reading comprehension, and translation. These tasks evaluated the model’s capabilities across different levels of language processing, thereby enabling a comprehensive assessment of the transliteration scheme’s effectiveness. Details regarding the pre-training data and parameter settings can be found in Appendix C.

We designed the following comparative experiments:

• Direct Continued Pre-training: Continuing pre-training on the original model using raw corpora.

• Vocabulary Expansion: Augmenting the original model’s vocabulary with dedicated lexicons for each low-resource language.

• UROMAN Transliteration: Applying universal romanization tools for transliteration.

• Three Progressive Transliteration Strategies: As detailed in Section 3.4.

4.2 Text Classification

We first evaluated the effectiveness of various transliteration schemes on the text classification task. The experiments utilized the WCM-v2 dataset
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Low-resource Languages (Acc / F1)</th>
    <th colspan="2">Chinese</th>
    <th colspan="2">Average</th>
  </tr>
  <tr>
    <th>bo</th>
    <th>mn</th>
    <th>ug</th>
    <th>Acc</th>
    <th>F1</th>
    <th>Minorities</th>
    <th>All</th>
  </tr>
  <tr>
    <td>Base Llama2</td>
    <td>28.65 / 21.23</td>
    <td>1.78 / 1.65</td>
    <td>73.33 / 74.69</td>
    <td>86.12</td>
    <td>85.91</td>
    <td>13.48 / 9.01</td>
    <td>48.15 / 48.78</td>
  </tr>
  <tr>
    <td>Expanded Vocab</td>
    <td><b>53.96</b> / 51.69</td>
    <td>64.45 / <b>67.95</b></td>
    <td>76.00 / 82.91</td>
    <td>89.95</td>
    <td>89.91</td>
    <td><b>62.58</b> / <b>64.56</b></td>
    <td><b>75.64</b> / <b>76.34</b></td>
  </tr>
  <tr>
    <td>UROMAN</td>
    <td>49.37 / 48.88</td>
    <td>64.92 / 67.78</td>
    <td><b>81.33</b> / <b>86.18</b></td>
    <td>89.82</td>
    <td>89.75</td>
    <td>62.10 / 63.47</td>
    <td>75.33 / 75.84</td>
  </tr>
  <tr>
    <td>Basic Trans.</td>
    <td>52.16 / 52.18</td>
    <td><b>66.46</b> / <b>69.55</b></td>
    <td>74.67 / 81.51</td>
    <td>89.92</td>
    <td>89.81</td>
    <td><b>63.40</b> / <b>65.17</b></td>
    <td><b>76.06</b> / <b>76.60</b></td>
  </tr>
  <tr>
    <td>Token-Opt Trans.</td>
    <td><b>54.14</b> / <b>54.94</b></td>
    <td>61.25 / 64.61</td>
    <td>81.00 / 85.31</td>
    <td><b>90.15</b></td>
    <td><b>90.07</b></td>
    <td>60.80 / 63.54</td>
    <td>74.81 / 75.86</td>
  </tr>
  <tr>
    <td>Hybrid Trans.</td>
    <td>50.45 / 52.58</td>
    <td>61.25 / 65.15</td>
    <td>65.33 / 75.59</td>
    <td>90.00</td>
    <td>89.95</td>
    <td>58.80 / 62.15</td>
    <td>73.68 / 75.02</td>
  </tr>
</table>

Table 5: Performance comparison on the WCM-v2 dataset. The best scores are in bold, with the second best underlined. Base Llama2: directly fine-tuned on original texts; Expanded Vocab: vocabulary expansion for each low-resource language; Basic/Token-Opt/Hybrid Trans.: three progressive transliteration strategies. Minorities average is calculated as the mean of scores for low-resource languages.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">CMRC-Trained</th>
    <th colspan="4">SQuAD-Trained</th>
  </tr>
  <tr>
    <th colspan="2">Chinese</th>
    <th colspan="2">Tibetan</th>
    <th colspan="2">English</th>
    <th colspan="2">Tibetan</th>
  </tr>
  <tr>
    <th></th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Base Llama2</td>
    <td>77.2</td>
    <td>89.5</td>
    <td>7.9</td>
    <td>45.8</td>
    <td>89.5</td>
    <td>95.3</td>
    <td>6.5</td>
    <td>50.8</td>
  </tr>
  <tr>
    <td>Expanded Vocab</td>
    <td>81.3</td>
    <td>91.1</td>
    <td>11.5</td>
    <td>50.6</td>
    <td><b>89.9</b></td>
    <td><b>95.7</b></td>
    <td>10.3</td>
    <td>58.9</td>
  </tr>
  <tr>
    <td>UROMAN</td>
    <td>79.6</td>
    <td>88.3</td>
    <td>12.0</td>
    <td>53.4</td>
    <td>84.2</td>
    <td>88.7</td>
    <td>11.0</td>
    <td>61.1</td>
  </tr>
  <tr>
    <td>Basic Trans.</td>
    <td>87.7</td>
    <td>92.7</td>
    <td>15.5</td>
    <td>59.5</td>
    <td>87.7</td>
    <td>89.1</td>
    <td>12.7</td>
    <td>65.1</td>
  </tr>
  <tr>
    <td>Token-Opt Trans.</td>
    <td><b>88.4</b></td>
    <td><b>93.6</b></td>
    <td><b>16.0</b></td>
    <td><b>60.2</b></td>
    <td>88.0</td>
    <td>89.3</td>
    <td><b>13.5</b></td>
    <td><b>66.5</b></td>
  </tr>
  <tr>
    <td>Hybrid Trans.</td>
    <td>83.1</td>
    <td>90.2</td>
    <td>14.8</td>
    <td>58.8</td>
    <td>87.2</td>
    <td>89.0</td>
    <td>12.3</td>
    <td>64.9</td>
  </tr>
</table>

Table 6: Machine Reading Comprehension performance comparison. The best scores are in bold, with the second best underlined. Results show both source language (Chinese/English) and target language (Tibetan) performance under different training settings. EM: Exact Match score; F1: F1 score.

(see Appendix D.1), a classification dataset encompassing multiple ethnic minority languages of China (Yang et al., 2022). This dataset maintains balanced distributions across both categories and languages, containing texts from 10 domains including arts, geography, and history. The experimental results are shown in Table 5. To comprehensively evaluate the effectiveness of each approach, we focus not only on the overall performance but also specifically on the average performance across low-resource languages.

4.3 Machine Reading Comprehension

For the machine reading comprehension task, we evaluate the models’ performance by fine-tuning them on the Chinese CMRC dataset (Cui et al., 2019) and English SQuAD dataset (Rajpurkar, 2016), followed by zero-shot testing on the TibetanQA dataset (Sun et al., 2021) to assess their cross-lingual transfer capabilities. We also report the performance on the source languages to verify that our approaches maintain strong performance on high-resource languages while enabling effective cross-lingual transfer. The results are shown in Table 6. For detailed information about the datasets, please refer to Appendix D.2.

4.4 Machine Translation

To evaluate the models’ machine translation capabilities for low-resource languages, we conduct experiments on Chinese-to-Tibetan (zh-bo) and Chinese-to-Uyghur (zh-ug) translation tasks using the Flores-200 dataset (Costa-jussà et al., 2022). We employ few-shot prompting with three carefully selected examples for each language pair, ensuring the examples cover diverse linguistic patterns. The evaluation uses three standard metrics: BLEU score for overall translation quality, chrF for character-level accuracy, and Translation Edit Rate (TER) for measuring the amount of editing required to match the reference translation. Table 7 presents the results of our comparative evaluation. For detailed information about the dataset and prompts used, please refer to Appendix E.

4.5 Overall Analysis

Cross-task Performance Analysis Through a comparative analysis of experimental results across text classification, machine reading comprehension, and machine translation tasks, our proposed transliteration approach demonstrated excellent cross-lingual transfer capabilities. In text classification tasks, the basic transliteration strategy
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Chinese-to-Tibetan (zh-bo)</th>
    <th colspan="3">Chinese-to-Uyghur (zh-ug)</th>
  </tr>
  <tr>
    <th>BLEU↑</th>
    <th>chrF↑</th>
    <th>TER↓</th>
    <th>BLEU↑</th>
    <th>chrF↑</th>
    <th>TER↓</th>
  </tr>
  <tr>
    <td>Base Llama2</td>
    <td>3.5</td>
    <td>0.28</td>
    <td>0.92</td>
    <td>4.2</td>
    <td>0.31</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Expanded Vocab</td>
    <td>5.0</td>
    <td>0.35</td>
    <td>0.86</td>
    <td>5.7</td>
    <td>0.38</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>UROMAN</td>
    <td>4.5</td>
    <td>0.33</td>
    <td>0.88</td>
    <td>5.2</td>
    <td>0.36</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>Basic Trans.</td>
    <td><b>5.7</b></td>
    <td><b>0.37</b></td>
    <td><b>0.84</b></td>
    <td><u>6.4</u></td>
    <td><u>0.40</u></td>
    <td><u>0.81</u></td>
  </tr>
  <tr>
    <td>Token-Opt Trans.</td>
    <td><b>6.3</b></td>
    <td><b>0.39</b></td>
    <td><b>0.82</b></td>
    <td><b>7.0</b></td>
    <td><b>0.42</b></td>
    <td><b>0.79</b></td>
  </tr>
  <tr>
    <td>Hybrid Trans.</td>
    <td>3.8</td>
    <td>0.30</td>
    <td>0.90</td>
    <td>4.5</td>
    <td>0.33</td>
    <td>0.87</td>
  </tr>
</table>

Table 7: Machine Translation performance comparison on Flores-200 dataset using few-shot prompting (3 examples).
↑: higher is better, ↓: lower is better. The best scores are in bold, with the second best underlined. TER: Translation Edit Rate.

achieved an average accuracy of 63.40% on low-resource languages, showing an improvement of 0.82% compared to the vocabulary expansion approach. For machine reading comprehension tasks, the tokenizer-optimized transliteration strategy achieved an exact match score of 16.0% on Chinese-to-Tibetan transfer, outperforming the vocabulary expansion approach by 4.5%. This strategy also exhibited superior performance in translation tasks, achieving a BLEU score of 6.3 in Chinese-to-Tibetan translation. Notably, these improvements were achieved while maintaining high performance on resource-rich languages, as exemplified by our approach achieving 90.15% accuracy on Chinese text classification tasks.

Key Findings Our experimental results yield three significant findings:

• Performance and Efficiency: Our transliteration approaches consistently outperformed traditional vocabulary expansion methods across tasks, with both basic and tokenizer-optimized strategies showing exceptional results. By leveraging existing tokenizer characteristics, these approaches significantly improved low-resource language processing without vocabulary expansion.

• Untapped Potential: Despite using simple frequency-based encoding schemes (B-X, Aa-Fk) and random token assignments, our methods demonstrated remarkable effectiveness. This suggests substantial room for improvement through the incorporation of linguistic features and more sophisticated encoding strategies.

• Scalable Framework: Our findings establish a new paradigm for low-resource language processing, offering a more promising direction than vocabulary expansion. The success of this relatively simple implementation particularly demonstrates its potential for scaling to multiple low-resource languages.

These results not only validate our approach but also indicate that more sophisticated versions of these strategies could yield even more significant improvements in low-resource language processing.

5 Conclusion

In this paper, we introduce a novel Huffman-based transliteration framework that addresses three critical challenges in low-resource language processing: cross-lingual transfer, storage efficiency, and practical deployment. Our framework demonstrates superior performance across diverse tasks while maintaining a lightweight implementation. The basic and tokenizer-optimized strategies consistently outperform traditional approaches, achieving up to 4.5% improvement in cross-lingual machine reading comprehension and significant gains in translation tasks, all while preserving performance on high-resource languages. Beyond performance gains, our approach offers unique advantages in compression efficiency, reducing both file size and token count by 2-3 times without sacrificing reversibility. Most importantly, our framework’s success with simple frequency-based encoding suggests substantial potential for improvement through the incorporation of linguistic features and more sophisticated encoding strategies. These findings establish a promising direction for scaling language technologies to the world’s many low-resource languages, offering a more practical alternative to the traditional vocabulary expansion paradigm.
Limitations

While our approach demonstrates promising results, there are several important limitations to consider. Our current evaluation scope is restricted to three low-resource languages with non-Latin scripts. Although the framework is theoretically extensible to other writing systems, specific adaptations may be necessary to accommodate their unique characteristics. The limited availability of evaluation datasets for low-resource languages also poses a challenge, particularly in tasks like machine reading comprehension, where we could only assess performance on a subset of languages.

From a practical perspective, our approach faces a trade-off between storage efficiency and computational overhead. While we achieve significant reductions in storage requirements, the transliteration and restoration processes introduce additional computational steps that could impact real-time performance, especially in scenarios requiring frequent language switching. Furthermore, our current encoding scheme relies primarily on character frequency, leaving room for potential improvements through the incorporation of linguistic features such as phonemes and morphological information.

Acknowledgments

This work is supported by the National Social Science Foundation (22&ZD035), the National Nature Science Foundation (61972436), and the Minzu University of China Foundation (GRSCP202316, 2023QNYL22, 2024GJYY43).

References

Chantal Amrhein and Rico Sennrich. 2020. On romanization for model transfer between scripts in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2461–2469.

Antonios Anastasopoulos and Graham Neubig. 2019. Pushing the limits of low-resource morphological inflection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 984–996.

A Conneau. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.

Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.

Emilio Cueva, Adrian López-Monroy, Fernando Sánchez Vega, and Thamar Solorio. 2024. Adaptive cross-lingual text classification through in-context one-shot demonstrations. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8317–8335.

Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. A span-extraction dataset for chinese machine reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5883–5889.

Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177.

Ameet Deshpande, Partha Talukdar, and Karthik Narasimhan. 2021. When is bert multilingual? isolating crucial ingredients for cross-lingual transfer. arXiv preprint arXiv:2110.14782.

Tejas Dhamecha, Rudra Murthy, Samarth Bharadwaj, Karthik Sankaranarayanan, and Pushpak Bhattacharyya. 2021. Role of language relatedness in multilingual fine-tuning of language models: A case study in indo-aryan languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8584–8595.

Wen Ding, Fei Jia, Hainan Xu, Yu Xi, Junjie Lai, and Boris Ginsburg. 2024. Romanization encoding for multilingual asr. arXiv preprint arXiv:2407.04368.

Changjiang Gao, Hongda Hu, Peng Hu, Jiajun Chen, Jixing Li, and Shujian Huang. 2024. Multilingual pre-training and instruction tuning improve cross-lingual knowledge alignment, but only shallowly. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6101–6117.

Ulf Hermjakob, Jonathan May, and Kevin Knight. 2018. Out-of-the-box universal romanization tool uroman. In Proceedings of ACL 2018, system demonstrations, pages 13–18.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293.
Sunkyong Kim, Dayeon Ki, Yireun Kim, and Jinsik Lee. Translating qa is enough: A key to unlocking in-context cross-lingual performance. In ICML 2024 Workshop on In-Context Learning.

T Kudo. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226.

Linlin Liu, Bosheng Ding, Lidong Bing, Shafiq Joty, Luo Si, and Chunyan Miao. 2021. Mulda: A multilingual data augmentation framework for low-resource cross-lingual ner. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5834–5846.

Yihong Liu, Chunlan Ma, Haotian Ye, and Hinrich Schütze. 2024a. Translico: A contrastive learning framework to address the script barrier in multilingual pretrained language models. arXiv preprint arXiv:2401.06620.

Yihong Liu, Chunlan Ma, Haotian Ye, and Hinrich Schütze. 2024b. Transmi: A framework to create strong baselines from multilingual pretrained language models for transliterated data. arXiv preprint arXiv:2405.09913.

Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Ayooob ImaniGooghari, Orgest Xheli, Haotian Ye, Chunlan Ma, François Yvon, and Hinrich Schütze. 2025. How transliterations improve crosslingual alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2417–2433.

Chunlan Ma, Yihong Liu, Haotian Ye, and Hinrich Schütze. 2024. Exploring the role of transliteration in in-context learning for low-resource languages written in non-latin scripts. arXiv preprint arXiv:2407.02320.

Ibraheem Muhammad Moosa, Mahmud Elahi Akhter, and Ashfia Binte Habib. 2023. Does transliteration help multilingual language modeling? In Findings of the Association for Computational Linguistics: EACL 2023, pages 670–685.

Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mbert is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448–462.

Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 116–126.

T Pires. 2019. How multilingual is multilingual bert. arXiv preprint arXiv:1906.01502.

Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2024. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97):1–52.

Sukanya Purkayastha, Sebastian Ruder, Jonas Pfeiffer, Iryna Gurevych, and Ivan Vulić. 2023. Romanization-based large-scale adaptation of multilingual language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7996–8005.

Jirui Qi, Raquel Fernández, and Arianna Bisazza. 2023. Cross-lingual consistency of factual knowledge in multilingual language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10650–10666.

P Rajpurkar. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.

Rico Sennrich. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.

Y Sun, S Liu, C Chen, Z Dan, and X Zhao. 2021. Construction of high-quality tibetan dataset for machine reading comprehension. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 208–218.

Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tanmoy Chakraborty. 2023. Multilingual llms are better cross-lingual in-context learners with alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6292–6307.

Mingxu Tao, Chen Zhang, Quzhe Huang, Tianyao Ma, Songfang Huang, Dongyan Zhao, and Yansong Feng. 2024. Unlocking the potential of model merging for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8705–8720.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Hans H Wellisch. 1978. The conversion of scripts-its nature, history, and utilization.

Zhuang Wenhao, Sun Yuan, and Zhao Xiaobing. 2024. Tilamb: (tilamb: A tibetan large language model based on incremental pre-training). In Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference), pages 254–267.
Shaoyang Xu, Junzhuo Li, and Deyi Xiong. 2023. Language representation projection: Can we transfer factual knowledge across languages in multilingual language models? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3692–3702.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Ziqing Yang, Zihang Xu, Yiming Cui, Baoxin Wang, Min Lin, Dayong Wu, and Zhigang Chen. 2022. Cino: A chinese minority pre-trained language model. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3937–3949.

Jiacheng Ye, Xijia Tao, and Lingpeng Kong. 2023. Language versatilists vs. specialists: An empirical revisiting on multilingual transfer ability. arXiv preprint arXiv:2306.06688.

Wenhao Zhuang and Yuan Sun. 2025. Cute: A multilingual dataset for enhancing cross-lingual knowledge transfer in low-resource languages. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10037–10046.
A Auxiliary Models

To achieve a comprehensive end-to-end system, we developed three auxiliary models for input language identification, output language identification, and Chinese pinyin conversion. These models collectively form a complete language processing pipeline, ensuring that the system can accurately process multilingual inputs and generate appropriate outputs.

A.1 Input Language Classifier

We trained specialized language identification models based on the FastText framework to accurately identify the language type of input text. The classifier supports five language categories: Tibetan (bo), Mongolian (mn), Uyghur (ug), Chinese (zh), and other languages (other). The training data was sourced from multiple datasets, including CUTE (Zhuang and Sun, 2025), WCM-v2 (Yang et al., 2022), and other open-source datasets, to ensure the model can process text from various domains. The training parameters for the input language classifier are shown in Table 8.

The evaluation results on the test set of 5,000 entries show that the classifier achieved a high level of classification across all languages and can be used for actual classification needs. The evaluation results are shown in Table 9.

<table>
  <tr>
    <th>Language</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Tibetan</td>
    <td>0.992</td>
    <td>0.989</td>
    <td>0.991</td>
  </tr>
  <tr>
    <td>Mongolian</td>
    <td>0.987</td>
    <td>0.985</td>
    <td>0.986</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>0.995</td>
    <td>0.993</td>
    <td>0.994</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>0.998</td>
    <td>0.997</td>
    <td>0.998</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>0.981</td>
    <td>0.978</td>
    <td>0.980</td>
  </tr>
</table>

Table 9: Performance of the input language classifier on various languages.

A.2 Transliterated Text Classifier

To accurately identify transliterated text in model outputs and guide proper language restoration, we trained a specialized FastText classifier. The distinguishing feature of this classifier lies in its need to process transliterated text; therefore, we utilized parallel corpora of transliterated text for training, ensuring the model could recognize textual features under different transliteration strategies. The training parameters for the output language classifier are shown in Table 8.

The classification performance on the transliterated text test set is shown in Table 10.

<table>
  <tr>
    <th>Language</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Tibetan</td>
    <td>0.988</td>
    <td>0.985</td>
    <td>0.987</td>
  </tr>
  <tr>
    <td>Mongolian</td>
    <td>0.983</td>
    <td>0.981</td>
    <td>0.982</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>0.991</td>
    <td>0.989</td>
    <td>0.990</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>0.995</td>
    <td>0.994</td>
    <td>0.995</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>0.992</td>
    <td>0.990</td>
    <td>0.991</td>
  </tr>
</table>

Table 10: Performance of the transliteration text classifier in various languages.

A.3 Pinyin-to-Chinese Converter

For Chinese pinyin conversion, we performed task-specific fine-tuning based on the Qwen2.5-0.5B model (Yang et al., 2024). This model handles the conversion from pinyin sequences to Chinese characters, which is a typical sequence-to-sequence conversion task. We used approximately 1 million pinyin-character pairs for training, with data sourced from news texts, Wikipedia, and general domain texts. The parameters used for fine-tuning are shown in Table 11.

<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>128</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>2e-5</td>
  </tr>
  <tr>
    <td>Max Length</td>
    <td>2048</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Warmup Steps</td>
    <td>1000</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>0.01</td>
  </tr>
</table>

Table 11: Qwen2.5-0.5B fine-tuning parameter settings.

The performance evaluation of the model on the test set is shown in Table 12.

These three auxiliary models collectively form a complete language processing pipeline, capable of accurately identifying input languages, processing transliterated text, and converting pinyin to Chinese characters when needed. In practical applications, these models have demonstrated stable performance and high accuracy.

B Compression Analysis

To comprehensively evaluate different transliteration approaches, we compare our three strategies with two baselines: vocabulary expansion (adding 6,000 tokens for each low-resource language) and UROMAN (a widely-used romaniza-
<table>
  <tr>
    <th>Parameter</th>
    <th>Input Classifier</th>
    <th>Output Classifier</th>
    <th>Note</th>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>0.1</td>
    <td>0.05</td>
    <td>Initial learning rate</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>25</td>
    <td>30</td>
    <td>Training epochs</td>
  </tr>
  <tr>
    <td>Word n-grams</td>
    <td>2</td>
    <td>3</td>
    <td>Maximum length of word n-gram</td>
  </tr>
  <tr>
    <td>Vector Dimension</td>
    <td>100</td>
    <td>150</td>
    <td>Embedding dimension</td>
  </tr>
  <tr>
    <td>Context Window</td>
    <td>5</td>
    <td>7</td>
    <td>Size of context window</td>
  </tr>
  <tr>
    <td>Min Word Count</td>
    <td>5</td>
    <td>3</td>
    <td>Minimum word frequency</td>
  </tr>
</table>

Table 8: Training parameters for FastText language classifiers. The input classifier is optimized for raw text classification, while the output classifier is specifically tuned for transliterated text patterns with slightly different hyperparameters.

<table>
  <tr>
    <th>Metric</th>
    <th>Value</th>
    <th>Note</th>
  </tr>
  <tr>
    <td>Character Accuracy</td>
    <td>0.975</td>
    <td>Single character accuracy</td>
  </tr>
  <tr>
    <td>Sentence Accuracy</td>
    <td>0.892</td>
    <td>Complete sentence accuracy</td>
  </tr>
  <tr>
    <td>BLEU Score</td>
    <td>96.8</td>
    <td>Overall translation quality</td>
  </tr>
  <tr>
    <td>Inference Speed</td>
    <td>125ms/sent</td>
    <td>Average processing time</td>
  </tr>
</table>

Table 12: Evaluation of Pinyin Conversion Model Performance. The model shows strong performance in character-level accuracy and complete sentence conversion, with reasonable inference speed suitable for real-time applications.

tion tool). Table 13 presents the compression performance across different approaches and languages.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Lang</th>
    <th colspan="2">File Compr.</th>
    <th colspan="2">Token Compr.</th>
  </tr>
  <tr>
    <th>bo</th>
    <th>mn</th>
    <th>ug</th>
    <th>zh</th>
  </tr>
  <tr>
    <td rowspan="4">Vocab Expansion</td>
    <td>bo</td>
    <td>1.00x</td>
    <td>6.92x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>mn</td>
    <td>1.00x</td>
    <td>9.66x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ug</td>
    <td>1.00x</td>
    <td>4.49x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>zh</td>
    <td>1.00x</td>
    <td>1.75x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">UROMAN</td>
    <td>bo</td>
    <td>2.07x</td>
    <td>1.88x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>mn</td>
    <td>2.41x</td>
    <td>5.16x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ug</td>
    <td>1.78x</td>
    <td>2.46x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>zh</td>
    <td>1.00x</td>
    <td>1.12x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">Basic</td>
    <td>bo</td>
    <td>1.98x</td>
    <td>1.33x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>mn</td>
    <td>1.95x</td>
    <td>2.57x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ug</td>
    <td>1.25x</td>
    <td>1.00x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>zh</td>
    <td>0.73x</td>
    <td>0.90x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">Tokenizer</td>
    <td>bo</td>
    <td>2.61x</td>
    <td>1.80x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>mn</td>
    <td>2.07x</td>
    <td>3.85x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ug</td>
    <td>1.27x</td>
    <td>1.39x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>zh</td>
    <td>0.73x</td>
    <td>0.90x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">Hybrid</td>
    <td>bo</td>
    <td>2.61x</td>
    <td>2.23x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>mn</td>
    <td>2.07x</td>
    <td>4.98x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ug</td>
    <td>1.27x</td>
    <td>1.92x</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>zh</td>
    <td>0.73x</td>
    <td>1.41x</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 13: Compression performance across different approaches and languages. File Compr. shows the ratio of original file size to transliterated file size. Token Compr. indicates the ratio of original token count to transliterated token count using Llama2 tokenizer. Language codes: bo (Tibetan), mn (Mongolian), ug (Uyghur), zh (Chinese).

The vocabulary expansion approach achieves the highest token compression ratios but maintains original file sizes. UROMAN demonstrates good compression performance in both file size and token count. Our proposed methods show progressive improvements from Basic to Hybrid strategies, with the Hybrid approach achieving competitive token compression while maintaining strong file size reduction. Note that Chinese (zh) shows different patterns due to its unique characteristics in tokenization and encoding.

C Training Details

C.1 Pre-training Data

The statistics of the raw corpora used for pre-training are shown in Table 14. The data is primarily sourced from the CUTE parallel corpus (Zhuang and Sun, 2025), which provides high-quality aligned multilingual content across Chinese, Uyghur, Tibetan, and English languages. For Mongolian, we followed the data collection and quality assessment methodology described in the CUTE paper to ensure comparable data quality and distribution.
<table>
  <tr>
    <th>Language</th>
    <th>Lines</th>
    <th>Size (GB)</th>
  </tr>
  <tr>
    <td>Tibetan (bo)</td>
    <td>934,140</td>
    <td>11.22</td>
  </tr>
  <tr>
    <td>Mongolian (mn)</td>
    <td>933,941</td>
    <td>11.48</td>
  </tr>
  <tr>
    <td>Uyghur (ug)</td>
    <td>934,002</td>
    <td>7.37</td>
  </tr>
  <tr>
    <td>Chinese (zh)</td>
    <td>933,946</td>
    <td>2.54</td>
  </tr>
  <tr>
    <td>English (en)</td>
    <td>933,989</td>
    <td>3.60</td>
  </tr>
</table>

Table 14: Pre-training Corpora Statistics. The data is primarily sourced from the CUTE parallel corpus, with additional Mongolian data collected following similar quality standards.

C.2 Training Parameter Settings

Table 15 lists the main parameter settings for the pre-training and supervised fine-tuning phases.

D Dataset Details

D.1 WCM-v2 Dataset

WCM-v2 is a multilingual text classification dataset covering 10 domains including arts, geography, and history (Yang et al., 2022). The dataset is characterized by its balanced distribution across both categories and languages, containing Chinese training sets and test sets in multiple languages. Table 16 shows the sample distribution of each language across different categories.

D.2 Machine Reading Comprehension Datasets

We utilize three machine reading comprehension (MRC) datasets for evaluation. Table 17 shows the key statistics of these datasets.

<table>
  <tr>
    <th>Dataset</th>
    <th>Train</th>
    <th>Dev</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>SQuAD v1.1</td>
    <td>87,599</td>
    <td>10,570</td>
    <td>-</td>
  </tr>
  <tr>
    <td>CMRC 2018</td>
    <td>10,142</td>
    <td>3,219</td>
    <td>1,002</td>
  </tr>
  <tr>
    <td>TibetanQA</td>
    <td>-</td>
    <td>-</td>
    <td>2,007</td>
  </tr>
</table>

Table 17: Statistics of machine reading comprehension datasets used in our experiments. TibetanQA is used only for testing cross-lingual transfer capability.

SQuAD The Stanford Question Answering Dataset (SQuAD) v1.1 (Rajpurkar, 2016) is a widely used English reading comprehension dataset containing over 100,000 question-answer pairs. The questions and answers were created by crowdworkers based on Wikipedia articles, with answers being continuous spans from the corresponding reading passages.

CMRC The Chinese Machine Reading Comprehension (CMRC) 2018 dataset (Cui et al., 2019) follows a similar format to SQuAD, featuring span-extraction questions in Chinese. The dataset covers various domains, making it suitable for evaluating Chinese reading comprehension capabilities.

TibetanQA TibetanQA (Sun et al., 2021) is a Tibetan machine reading comprehension dataset, with 2,007 publicly released question-answer pairs for evaluation. While the full dataset contains 20,000 question-answer pairs annotated from articles on Tibetan web resources, only a portion is publicly available and used in our experiments for zero-shot cross-lingual evaluation.

Note on Language Coverage While our study aims to evaluate cross-lingual transfer across multiple low-resource languages, we were unable to identify suitable machine reading comprehension datasets for Mongolian and Uyghur languages at the time of our research. This limitation highlights the scarcity of evaluation resources for these languages in certain NLP tasks.

E Translation Details

Flores-200 Dataset The Flores-200 dataset is a multilingual benchmark for evaluating machine translation systems, encompassing 200 languages (Costa-jussà et al., 2022). The sentences in the dataset are derived from English Wikipedia articles and professionally translated into other languages. We utilize both the development and test sets, which contain 997 and 1,012 samples per language pair, respectively. To ensure fair evaluation, we conduct our experiments exclusively on the test set.

Translation Prompts We employ English as the unified instruction language for translation tasks. Each prompt contains three carefully selected translation examples (3-shot), with low-resource language text appearing only in the source-target translation pairs. Specifically, the prompt template follows this structure:

(1) An instruction header:

Translate the following Chinese text to {target_language}

(2) Three example translation pairs, each formatted as:
<table>
  <tr>
    <th>Hyperparameter</th>
    <th>Pre-training</th>
    <th>Fine-tuning</th>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>1.0e-4</td>
    <td>2.0e-5</td>
  </tr>
  <tr>
    <td>Training Epochs</td>
    <td>1.0</td>
    <td>3.0</td>
  </tr>
  <tr>
    <td>Global Batch Size</td>
    <td>1024</td>
    <td>256</td>
  </tr>
  <tr>
    <td>Max Sequence Length</td>
    <td>4096</td>
    <td>4096</td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0.05</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>Data Type</td>
    <td>BF16</td>
    <td>BF16</td>
  </tr>
  <tr>
    <td>LR Scheduler</td>
    <td>Cosine</td>
    <td>Cosine</td>
  </tr>
</table>

Table 15: Hyperparameter settings for pre-training and supervised fine-tuning phases. During the pre-training phase, except for vocabulary expansion, we observed frequent loss spike phenomena when the learning rate was set to 2.0e-4. After reducing it to 1.0e-4, the training process became more stable.

<table>
  <tr>
    <th>Category</th>
    <th>mn</th>
    <th>bo</th>
    <th>ug</th>
    <th>zh-train</th>
    <th>zh-test</th>
  </tr>
  <tr>
    <td>Arts</td>
    <td>135</td>
    <td>141</td>
    <td>3</td>
    <td>2,657</td>
    <td>335</td>
  </tr>
  <tr>
    <td>Geography</td>
    <td>76</td>
    <td>339</td>
    <td>256</td>
    <td>12,854</td>
    <td>1,644</td>
  </tr>
  <tr>
    <td>History</td>
    <td>66</td>
    <td>111</td>
    <td>0</td>
    <td>1,771</td>
    <td>248</td>
  </tr>
  <tr>
    <td>Nature</td>
    <td>7</td>
    <td>0</td>
    <td>7</td>
    <td>1,105</td>
    <td>110</td>
  </tr>
  <tr>
    <td>Natural Science</td>
    <td>779</td>
    <td>133</td>
    <td>20</td>
    <td>2,314</td>
    <td>287</td>
  </tr>
  <tr>
    <td>People</td>
    <td>1,402</td>
    <td>111</td>
    <td>0</td>
    <td>7,706</td>
    <td>924</td>
  </tr>
  <tr>
    <td>Technology</td>
    <td>191</td>
    <td>163</td>
    <td>8</td>
    <td>1,184</td>
    <td>152</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>6</td>
    <td>1</td>
    <td>0</td>
    <td>936</td>
    <td>118</td>
  </tr>
  <tr>
    <td>Economy</td>
    <td>205</td>
    <td>0</td>
    <td>0</td>
    <td>922</td>
    <td>109</td>
  </tr>
  <tr>
    <td>Health</td>
    <td>106</td>
    <td>111</td>
    <td>6</td>
    <td>551</td>
    <td>73</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>2,973</th>
    <th>1,110</th>
    <th>300</th>
    <th>32,000</th>
    <th>3,995</th>
  </tr>
</table>

Table 16: Sample distribution across categories and languages in the WCM-v2 dataset. The dataset contains training and test sets for Chinese (zh), and test sets for ethnic minority languages (mn: Mongolian, bo: Tibetan, ug: Uyghur). Additional test sets for Korean, Kazakh, and Kyrgyz are also available in the dataset but not used in our experiments.

Chinese: [source text]
{target_language}: [translation]

(3) The translation request:

    Now translate this:
    Chinese: [input text]

This design is motivated by two key considerations: First, utilizing English as the instruction language leverages the model’s strong capabilities in English; Second, by minimizing the presence of low-resource languages in the prompt, we can better evaluate the model’s genuine translation capabilities rather than simple pattern matching. The three examples are selected to cover diverse sentence structures and vocabulary complexity, helping the model understand the requirements of the translation task.

Reviewer paper summaries: This study uses Huffman coding to (1) create a reversible transliteration method that lets people accurately recover the original script after converting to Latin letters; (2) reduce storage space during transliteration, which helps with both data handling and training efficiency.
Summary of strengths: 1. The use of a Huffman coding-based transliteration scheme for low-resource languages is a novel contribution. The authors also present an end-to-end framework that integrates FastText-based language identification.
2. The study very clearly outlines the three-stage approach that they took in Figure 1.
Summary of weaknesses: 1. Introduction: I think a specific example would be helpful to explain the idea of transliteration.
2. Because the authors relax the prefix property (i.e., they no longer fully follow the original Huffman rule as described in Section 3.3), decoding may not be as robust or error-tolerant, especially if the structure gets disrupted (e.g., by typos, formatting issues, or case sensitivity). Further explanations for potential drawback of using Huffman-based encoding design might be included in the manuscript.
Comments, suggestions and typos: 1. Several cited works are currently listed as arXiv preprints. If these papers have since been published in peer-reviewed venues (e.g., conferences or journals), I recommend updating the references to reflect their final publication details.
Overall assessment scores: 3.5
Metareview score: 4
Metareview: This paper aims to improve LLM performance on low-resource languages (Tibetan, Mongolian, Uyghur, and Chinese are tested) by transforming them all into a common, transliterated latin representation. The authors hypothesize that sharing a transliteration scheme will facilitate cross-lingual transfer as different languages will share tokens. The representation is constructed via a modified huffman-coding, and results in significant text compression (both in terms of raw data storage and LLM token counts). The scheme is also deterministically reversible. Continued LLM pre-training with huffman-transliterated data seems to improve performance on the test languages across a few benchmark tasks.
Summary of reasons to publish: The paper is clearly written.

The huffman-coding based transliteration scheme is novel and interesting, while achieving good results in the experiments presented.