Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation (2024)

Abstract: Subword tokenization is a common method for vocabulary building in Neural Machine Translation (NMT) models. However, increasingly complex tasks have revealed its disadvantages. First, a vocabulary cannot be modified once it is learned, making it hard to adapt to new words. Second, in multilingual translation, the imbalance in data volumes across different languages spreads to the vocabulary, exacerbating translations involving low-resource languages. While byte-based tokenization addresses these issues, byte-based models struggle with the low information density inherent in UTF-8 byte sequences. Previous works enhance token semantics through local contextualization but fail to select an appropriate contextualizing scope based on the input. Consequently, we propose the Multi-Scale Contextualization (MSC) method, which learns contextualized information of varying scales across different hidden state dimensions. It then leverages the attention module to dynamically integrate the multi-scale contextualized information. Experiments show that MSC significantly outperforms subwordbased and other byte-based methods in both multilingual and out-of-domain scenarios. We have uploaded the code to github1.

heading: 1 Introduction
text: In neural machine translation (NMT) systems, subword tokenization has been the most common and effective method to mitigate the out-of-vocabulary (OOV) problem. However, both BPE (Sennrich et al., 2016) and SentencePiece (Kudo and Richardson, 2018) fix the word segmentation rule or vocabulary once they have learned them on the initial corpus, making it difficult to ensure adaptation to new corpora. This is worsened in out-of-domain scenarios. Additionally, in multilingual scenarios
*Corresponding author. 1https://github.com/ictnlp/
Multiscale-Contextualization
with data imbalance, subword vocabularies tend to focus on high-resource languages, overlooking low-resource ones. This imbalance can cause an increase in OOV cases or over-segmentation of texts, harmful to translation model performance.
Byte-based method is able to solve these problems with few embedding parameters and has aroused extensive researches (Wang et al., 2020; Shaham and Levy, 2021; Xue et al., 2022; Yu et al., 2023; Edman et al., 2023; Sreedhar et al., 2023). In byte-based models, text is converted into byte sequences according to UTF-8 encoding, with each byte as a token within the vocabulary. They generally use a vocabulary with a maximum size of 256 but can adapt to imbalanced scenarios like multilingual translation and out-of-domain adaptation.
However, a feature of UTF-8 encoding hinders conventional Transformer model (Vaswani et al., 2017) from adapting well to byte-based vocabulary: a single character may correspond to 1 to 4 UTF-8 bytes. The number is 1 for English characters, but Arabic and many Asian languages require multiple bytes to represent a single character. Therefore, sometimes a single byte does not have a determined meaning; it requires the integration of local information to encode its semantics. To address that, various methods have been proposed for integrating local contextual information. SU4MT (Huang et al., 2023) learns contextual information with an Attentive Semantic Fusion layer, but requires accurate segmentation. MEGABYTE (Yu et al., 2023) segments a sentence into blocks of 4 and simply concatenates the tokens. Charformer (Tay et al., 2022) segments a sentence 4 times with block sizes ranging from 1 to 4 each, and employs mean-pooling to perform local integration. The weighted-sum of 4 results yields the final output. LOBEF (Sreedhar et al., 2023) proposed Byte-nCF, replacing mean-pooling with Convolutional Neural Networks (CNNs) for better performance.
Though these methods learn and leverage con-
1 9794
textual information in larger scales, they are limited by fixed block sizes and can not adjust the fusion weights according to the scripts of different languages. To remedy this, we propose the MultiScale Contextualization (MSC) method, which firstly learn contextual information of multiple scales in different hidden state dimensions, and secondly leverage the attention mechanism to fuse multi-scale information with dynamic weights. Our method better adapts to complex input scripts by allowing the model to adaptively fuse information of different granularities based on the varying content of the inputs. Experimental results demonstrate that our MSC method exhibits superior adaptability across various languages and datasets.

heading: 2 Method
text: In this section, we introduce the proposed MultiScale Contextualization (MSC) method. Bytebased models usually learn contextualized information implicitly. What MSC does is explicitly modeling the contextual information of multiple scales by grouping the hidden state dimensions and let different parts of the dimensions learn information of different scales.
Specifically, we insert a multi-scale contextualization module right before the Multi-Head Attention (MHA) module, as is depicted in Fig 1. The input vector x is divided according to the hidden state dimension into n parts [x1, x2, ..., xn].
Then, n contextualizing functions g(·) are applied to these parts respectively. A simple and effective structure for local contextualization is the
1-D convolution neural network (CNN). Therefore, we leverage CNNs with different kernel size k to control the contextualization scope. Since different dimensions are contextualized with various granularities, our model can realize multi-scale contextualization. To preserve the original information, g(·) is also allowed to be a "PassThrough" function, which directly returns the input without any operations.
gi(·, k) = { PassThrough(·) , k = 0 CNN(·, k) , k > 0 (1)
In equation (1), gi(·, k) means the contextualization function for group i, and k is the kernel size. Here, k=0 denotes the "PassThrough" function for simplification.
Next, the contextualized vector parts x̂i are calculated by gi(xi, k). Finally, they are concatenated to form x̂, which acts as the input of the MHA module.
Preliminary experiments on CNN configurations guided the best structure for the whole model. First, padding on the left side deteriorates model performance, so the CNNs are set to pad on both sides, as shown in Figure 1. Second, applying MSC to Transformer decoder layers causes a discrepancy between training and testing, when a token’s right side tokens are not yet generated. As a result, MSC is only applied to encoder layers.
It is worth noting that the kernel size k is recommended to be an odd number or zero, otherwise, manual zero-padding is required to keep the output length the same as the input. Empirically, it is
2
better to choose k from {0, 1, 3, 5, 7}.

heading: 3 Experiments
text: We experiment with two multilingual datasets and a domain-adaptation dataset to investigate the performances and properties of the MSC approach and other byte-based language models.

heading: 3.1 Datasets
text: Multilingual Many-to-One Translation
We use a multilingual TED corpus of 59 languages (Qi et al., 2018), TED-59, which includes both high and low-resource languages. All cases are English-centered. We collect the raw data from Salesky et al. (2023) and preprocess it with two subword-level vocabularies and a byte-level one. For subword-based baseline system, we leverage SentencePiece (Kudo and Richardson, 2018) and train a 32k vocabulary on the training set. We also incorporate the 250k mBART-50 (Liu et al., 2020; Tang et al., 2020) vocabulary for full lexical coverage. For Byte-level systems, we preprocess data with a 256 vocabulary using scripts2 from Shaham and Levy (2021).
Multilingual English-Centric Translation
We use the OPUS-7 corpus processed by Gu and Feng (2022), which is extracted from the OPUS100 corpus (Zhang et al., 2020). The OPUS-7 dataset contains a training corpus of 6 languages
2https://github.com/UriSha/EmbeddinglessNMT/ blob/master/embeddingless_scripts/byte_ preprocess.sh
(Ar, De, Fr, Nl, Ru, Zh) and their English Translations, with 1M sentences of each language.
Zero-shot Cross-domain Adaptation Besides multilingual scenarios, we also experiment with the zero-shot cross-domain adaptation ability of byte-based translation models with the WMT19 German→English (De→En) dataset. We train all models on the News domain and evaluate on test data from three domains used in Sreedhar et al. (2023) and Aharoni and Goldberg (2020) , which are Koran, IT, and Medical. We use the data preprocessed and provided by Sreedhar et al. (2023).

heading: 3.2 Models
text: We compare the proposed MSC approach mainly with other byte-based machine translation models.
• Transformer (Vaswani et al., 2017): The standard Transformer model without adaptations to byte sequences.
• Byte-nCF (Sreedhar et al., 2023): A strong byte-based model performing well under low-resource settings. The structure hyperparameters are of default setting3.
• MSC: We set n=8 in our experiments. The selection of k is discussed in Appendix A.
We also compare with subword-based methods.
• Learned: The standard Transformer model with a learned vocabulary.
3https://github.com/makeshn/LOBEF_Byte_NMT/ blob/main/embeddingless_scripts/train_byte_ncf. sh
3
• mBART: The standard Transformer model using the vocabulary of mBART (Liu et al., 2020). We do not use the pretrained checkpoint of mBART for fairness.
• Aharoni: The strongest baseline of subwordbased models in this parameter scale.
The other settings are discussed in Appendix B.

heading: 4 Results and Analyses

heading: 4.1 Multilingual Many-to-One Translation
text: Table 1 shows the results on TED-59 datasets. All byte-based methods are experimented three times to enhance the reliability of results. We report the average and standard deviation of the results. Aharoni et al. (2019) have selected four low-resource (LR) and four high-resource (HR) languages to show models’ performance on different training data scales, and we report results in the same way.
The average SacreBLEU scores of 58 translation directions (AVG-58) demonstrate that byte-based models are superior to subword-based models in massively multilingual scenario, despite of lower parameter usage.
Compared with other byte-based approaches, MSC performs better in almost all languages. While Byte-nCF learns a fixed set of combination weights of multi-scale contextual information for all languages, MSC adaptively leverages contextual information of different granularities at inference stage. For example, a single byte can represent a character or even a word in German, Italian, etc., so MSC leverages contextual information from its nearer neighborhood; a single byte may not be sufficient to form even a character, so MSC inclines to focus on contextual information of larger scales. We demonstrate this explanation later with an experiment in 4.4.

heading: 4.2 Multilingual English-Centric Translation
text: Table 2 shows the results on OPUS-7 dataset, which contains only seven high-resource languages. In this scenario, the subword-based model largely surpasses byte-based models. However, the performance gap is smaller when measured by COMET (Rei et al., 2022), a more reliable modelbased metric, as reported in Appendix C.
Among these byte-based models, MSC generally performs better than the others. To verify such improvements are not from randomness, we repeat them for three times and report the average and standard deviation of the results.

heading: 4.3 Zero-Shot Cross-Domain Adaptation
text: Table 3 shows the results on in-domain and zeroshot out-of-domain test datasets. For subwordbased models, using the dictionary trained on the News domain dataset to preprocess test sets from other domains results in a significant number of <unk> words. This leads to the model struggling to comprehend the input sentences and perform translations. However, byte-based models can eliminate the Out-Of-Vocabulary (OOV) issue, thereby achieving better performance in zero-shot translation scenarios. The results also demonstrate that our proposed MSC method has a significant advantage in zero-shot cross-domain adaptation.

heading: 4.4 Contextualization Scales
text: In section 2, we have introduced the hyperparameter k which controls the contextualization scale of our approach. Here, we experiment on TED-59 dataset to show how this modeling scale affect translation.
According to the Unicode rule, we group languages by the number of bytes they require to form a character, which are named "Byte-1", "Byte-2", and "Byte-3". Then, we select three languages for each group to represent that group, as listed below.
4
• Byte-1: French (Fr), German (De), Dutch (Nl)
• Byte-2: Russian (Ru), Thai (Th), Arabic (Ar)
• Byte-3: Chinese (Zh), Japanese (Ja), Korean (Ko)
For the selection of k series, which reflect the contextualization scales applied in a model, we experiment the small scales "0,0,1,1,3,3,5,5", the large scales "0,0,1,1,5,5,7,7", and the balanced scales "0,0,1,1,3,5,5,7". These models can leverage information of granularities of "1,3,5", "1,5,7", and "1,3,5,7" respectively4.
The results are exhibited in Table 4. First, the model of balanced scales performs the best averagely, because it is provided with contextualized information of more scales and has more options to choose. Second, if we ignore the balanced one and compare the other models, the performance is related to the language groups. The "∗" sign indicates the better performance between two models. For "Byte-1" group, a small scale is sufficient to model certain semantics, so the smaller scaled model performs better. For "Byte-3" groups, it requires larger contextual scales to form certain meanings, so the larger scaled model performs better.
4We have also experimented the granularities "9", "11" and "13", but they are harmful to translation quality. It demonstrates the model is unable to process information from too many tokens well.
These discoveries shade light on the selection of k series, which is the k series should be compatible with the language of input text.

heading: 5 Conclusions
text: In this paper, we make two primary contributions. Firstly, we show when byte-based models outperforms subword-based models and when they don’t. In massively multilingual translation scenarios which involves a wide range of languages, byte-based models exhibit clear superiority, particularly for low-resource languages. In limited language numbers with sufficient training data, bytebased models lags behind subword-based models. Secondly, we introduce a Multi-Scale Contextualization (MSC) method which enhances the adaptability of byte-based models to diverse inputs. By dynamically integrating multi-scale contextualized information, MSC outperforms other byte-based models in generally all languages.
Limitations
While our approach can adaptively integrate multiscale contextualized information, all contextualizing scopes k are predetermined. We will explore fully adaptive methods for multi-scale information extraction and integration for future work.

heading: Acknowledgements
text: We thank all the anonymous reviewers for their insightful and valuable comments. This paper is supported by National Natural Science Foundation of China (Grant No.62376260).

heading: A Selection of k Series
text: The selection of k varied across different languages. A critical determinant is the number of UTF-8 bytes required to represent a character. For languages that use the Latin alphabet, where a single byte can represent a character, a smaller k suffices; conversely, for languages where a character corresponds to multiple bytes, the information density of a single byte is lower, necessitating a larger k. Additionally, we find the "PassThrough(·)" function indispensable as it preserves the original information in the hidden states.
Empirically, the k series is "0,0,1,1,3,3,5,5" for De→En domain adaptation dataset, "0,0,1,1,3,5,5,7" for OPUS-7 dataset, and "0,0,3,3,5,5,7,7" for TED-59 dataset.

heading: B Detailed Model Settings
text: For a fair comparison, all model implementations are based on the Fairseq (Ott et al., 2019) codebase. In our experiments, all models contain 6 transformer encoder layers and 6 transformer decoder layers with 8 attention heads. The model dimension is 512 for word embedding and 2048 for feed-forward layers. The word embeddings of the encoder and decoder are shared for all models. For our method, the MSC module is applied to the first encoder layer. For Byte-nCF, we use the default model structure, but apply the shared-embedding setting, which on one hand prove to be better than its default settings, and on the other hand align with other experiments.
For model training, we train all models using 8 GPUs with batch size 8192 for each. We use adam optimizer (Kingma and Ba, 2015) with β=(0.9, 0.98) and 4k warm-up steps. The peak learning rate is 5e-4 in multilingual tasks and 7e-4 in De→En cross-domain adaptation task. Besides, we apply dropout 0.1 and label smoothing 0.1 for all models. We apply an early stop of 10, and average the last 5 checkpoints for evaluation. All models are evaluated using the SacreBLEU score.

heading: C Experiment Results Measured by COMET
text: 8

heading: Future Work
text: We will explore fully adaptive methods for multi-scale information extraction and integration for future work