Local Byte Fusion for Neural Machine Translation
Year: 2023

Abstract:
Subword tokenization schemes are the dominant technique used in current NLP models. However, such schemes can be rigid and tokenizers built on one corpus may not adapt well to other parallel corpora. It has also been observed that in multilingual corpora, subword tokenization schemes oversegment low-resource languages, leading to a drop in translation performance. An alternative to subword tokenizers is byte-based tokenization, i.e., tokenization into byte sequences using the UTF-8 encoding scheme. Byte tokens often represent inputs at a sub-character granularity, i.e., one character can be represented by a span of byte tokens. This results in much longer byte sequences that are hard to interpret without aggregating local information from multiple byte tokens. In this paper, we propose a Local Byte Fusion (LOBEF) method for byte-based machine translation—utilizing byte n-gram and word boundaries—to aggregate local semantic information. Extensive experiments on multilingual translation, zero-shot cross-lingual transfer, and domain adaptation reveal a consistent improvement over vanilla byte-based models. Further analysis also indicates that our byte-based models are parameter-efficient and perform competitive to subword models.

heading: 1 Introduction
text: Multilingual neural machine translation (NMT) has proven effective to transfer knowledge learned from a high-resource language to a low-resource language. However, existing multilingual NMT models still rely on a pre-built subword tokenizer (e.g., BPE (Sennrich et al., 2016), SentencePiece (Kudo and Richardson, 2018)) to tokenize a sentence into a sequence of subword units. This has two drawbacks. First, once the tokenizer is fixed, we lose the flexibility of changing the word tokenization if we aim to fine-tune the NMT model on another parallel corpus of interest for adaptation.
Second, when a subword tokenizer is built on unbalanced multilingual data, word tokens from a lowresource language are usually under-represented, resulting in over-segmentation of a word into many single characters. A recent study (Rust et al., 2021) measures over-segmentation by the fertility score of a subword scheme which indicates how many subwords a whole word is broken down into. It shows a negative correlation between the fertility score and the performance of multilingual models on the over-segmented languages.
Although character-based models have often been proposed as a solution to these problems (Gupta et al., 2019; Libovický and Fraser, 2020; Li et al., 2021), they come with their own tradeoffs related to the significant overhead of processing long character sequences during training and inference (Libovický and Fraser, 2020). Besides, these models still adopt a fixed vocabulary of characters, leading to the same issue as a fixed subword tokenizer for adaptation. Another point of difference is that in character-based methods, the vocabulary still consists of one unique embedding for each character under consideration. In byte-based approaches, the tokens are at a sub-character granularity and the model has to figure out how to combine bytes for different languages. Recently, there has been renewed interest in character-based models that adopt a byte tokenization scheme (Clark et al., 2021; Xue et al., 2021; Tay et al., 2021)—tokenization of texts into UTF-8 byte tokens. Although these byte-based models have shown competitive performance to subword models on multilingual NLU benchmarks (Hu et al., 2020), their performance on multilingual generation, especially on multilingual NMT is still underexplored. Although Shaham and Levy (2021) recently demonstrate the effectiveness of byte tokenization for bilingual machine translation, a comprehensive study of such bytebased methods on the multilingual paradigm across a wide variety of languages and domains is still
7199
missing. Particularly in the multilingual setting, as characters in different languages can be tokenized into a varying number of byte tokens, this produces byte sequences much longer than the original sentences, and vanilla byte-based MT models can only implicitly reconstruct character-/word-level representations from byte tokens in an entirely datadriven fashion.
To remedy these issues, we propose two variants of Local Byte Fusion (LOBEF) 1 techniques that explicitly aggregate byte tokens to learn character/word-level representations for byte-based NMT models. Our first variant utilizes four n-gram convolutional layers to aggregate bytes for learning character-level information, and our second variant utilizes word boundaries to aggregate a span of bytes for learning word-level context. We conduct extensive experiments to compare our methods with the vanilla byte-based model and the embeddingless model from Shaham and Levy (2021) in a multilingual translation setting. Our many-toone translation results show that aggregating local information in earlier layers encourages the model to capture local information for seven source languages, yielding an average gain of up to 1.4 BLEU over the vanilla byte-based NMT model while performing competitively with subword models. We further demonstrate the effectiveness of LOBEF on the zero-shot/few-shot cross-lingual transfer and cross-domain adaptation settings, showing the flexibility of byte-based NMT models over subword baselines when fine-tuning is required for data adaptation. Additionally, our method also improves over vanilla byte-based NMT models for adaptation. Our contributions are as follows:
• To the best of our knowledge, we are the first to evaluate byte-based embeddingless NMT models
1The code can be found at https://github.com/ makeshn/LOBEF_Byte_NMT
in a multilingual translation setting. • To further improve the encoding of local se-
mantics for byte-based NMT, we propose two variants of local fusion techniques based on character-/word-level aggregation over byte tokens.
• We provide a fine-grained analysis to show the effectiveness of byte-based models on crosslingual and domain adaptation settings.

heading: 2 Preliminaries

heading: 2.1 Unicode and UTF-8
text: Unicodeis a universal, platform-agnostic standard for handling text in most of the world’s writing systems, covering characters in all of the world’s living languages as well as emoji and non-visual codes. Each code point defined by Unicode is mapped to a unique integer, ranging from 0 to 10FFFF16. For instance, the English character set A-Z is denoted by the integers from 97-122. In modern computers, each Unicode code point can be implemented as bytes by multiple encoding protocols, and UTF-8 is the dominant encoding protocol used by over 95% of webpages.
In UTF-8, each Unicode code point is represented as one to four bytes (8 bits per byte) depending on the range of its Unicode integer. Some languages may have a combination of characters that require a varying number of bytes. For example, most characters in German require only a single byte, while some special characters like à or â use two bytes. Since the Unicode and the UTF-8 encoding scheme is already well-defined, we do not have to construct source and target vocabularies similar to how it is done for subword models. Tokenization and de-tokenization for byte based models is as simple as a single line of code in Python and does not involve any heuristic preprocessing. In this paper, we adopt the UTF-8 byte
tokens as inputs to our model.

heading: 2.2 Byte-based NMT
text: Shaham and Levy (2021) recently propose an embeddingless NMT model that takes sequences of UTF-8 byte tokens as the inputs and outputs, and uses a fixed one-hot representation for each byte token instead of a dense learnable embedding vector. Such a byte-based NMT model eliminates the input and output token embedding layers usually used in subword-based NMT models, leading to a significant reduction in model parameters.
Formally, given a source-target sequence pair from a parallel corpus (x, y) ∼ D where x = (x1, ..., xN ) and y = (y1, ..., yM ) are both sequences of byte tokens, the input sequence is first embedded by one-hot representations, i.e., Embed(x) = X ∈ RN×d, and further encoded into the source hidden representation Z by a vanilla L-layer Transformer encoder:
Z = Encoder(X, L). (1)
Finally, an attention-based decoder performs the attention over Z and estimates the probability of predicting the next byte token yt by
P (yt|y<t, x) = Decoder(y<t,Z). (2)
Compared to subword-based NMT models, bytebased NMT models have shown effectiveness on bilingual machine translation, while their performance in multilingual machine translation is still unexplored. Especially in the many-to-one translation, the encoder is used to encode multiple languages which aggregate varying numbers of byte tokens (i.e., 1 to 4 bytes) to represent one character.

heading: 3 Local Byte Fusion
text: For languages that do not exclusively use the English character set, encoding them often requires more than one byte. Vanilla byte-based models can only implicitly aggregate character-level or wordlevel representations for these languages, potentially resulting in poor interpretability and suboptimal results in multilingual settings. Hence, we propose two fusion techniques that encourage models to explicitly aggregate character-level and word-level information from byte sequences.
We also adopt byte sequences as inputs and outputs for our model, and use vanilla Transformer as the backbone. As we focus on multilingual encoding in this work, we only modify the encoder,
and adopt the same decoder architecture from (Shaham and Levy, 2021). Note that a more sophisticated design of the decoder will also involve a special design of decoding algorithms (Libovický et al., 2022) which goes beyond the scope of this work. Besides, to conduct a more comprehensive study, we also consider the case where we retain embedding layers for the encoder and decoder of the byte-based model. This implies that instead of one-hot representations for the byte sequences, we can learn dense vector representations. Since the vocabulary size of all byte tokens is 256, this does not amount to adding a significant number of extra parameters.
3.1 n-gram Convolutional Fusion (nCF)
Before we explicitly aggregate the character-level information, we first encode the input byte sequence by a shallow encoder with Ls Transformer layers, which allows the model to have a shallow access to the sentence context before local fusion.
S = Encoder(X, Ls) (3)
Since characters can be represented as a combination of 1 to 4 bytes depending on the languages, we apply four different 1-D convolutional layers to aggregate the n-gram byte tokens where n ∈ {1, 2, 3, 4}. Specifically, we define convn as the 1-D convolution layer with a kernel of size n and stride n. We do right padding at the end of the byte sequence. Therefore, when we use a stride n greater than 1, the length of the input byte sequence is reduced by a factor corresponding to the stride n. We then define the output from the convn layer by:
[ fn1 , · · · , fnN
n
] ∈ RNn ×d ← convn (S) . (4)
To make all of the outputs the same length as the input, we repeat the output tokens in place by a factor corresponding to the stride length n.
Fn = [ repeat(fn1 , n), · · · , repeat ( fnN n , n )] , (5)
where repeat(x, n) creates n copies of a vector x. Applying this repetition process to the output from each of the convolution layers, we have four representations of equal sequence length as the source sequence,2 i.e., F1,F2,F3,F4 ∈ RN×d. We pass these representations through a linear layer to get a
2Extra tokens at the end are truncated to ensure equal length.
single weighted representation:
F = 4∑
n=1
λnFn, (6)
where λ = [λ1, · · · , λ4] are weights for the n-gram representations. We pass this weighted representation to the remaining (L− Ls) Transformer layers to obtain the final encoder hidden representation which is further sent to the decoder by Eq. (2).
Z = Encoder (F, L− Ls) (7)
The n-gram fusion enables the model to learn what combination of the input byte sequence representations results in better character-level features.

heading: 3.2 Word-based Self-attention Fusion (WSF)
text: In addition, we also propose a word-based selfattention fusion method that utilizes the word boundary information in the raw sentence to aggregate byte tokens within the same word. As characters in most languages are represented by more than one byte and words contain varying number of characters, using byte tokens as input to the model results in a much longer sequence. Therefore, this property may require the model to recognize a meaningful span of byte tokens in order to capture the semantic of a word token in the raw sentence. However, vanilla byte-based NMT models (§2.2) use the traditional full self-attention, which implies that every byte token in the sequence attends to all byte tokens even though some far-away byte tokens may have little association to the query byte. Besides, as words are represented by a span of bytes in a small vocabulary of size 256, it is likely to produce a high attention weight between two identical byte tokens even when these byte tokens are used in two completely irrelevant words.
We tackle this issue by aggregating local information of a byte span for a word using a blockwise self-attention. Formally, for a byte sequence x = (x1, · · · , xN ), we define its (untokenized) word sequence as w = (w1, · · · , wT ) and a mapping π : [t] → [a : b] that maps the word index t to the beginning and the end indices of the corresponding byte span, i.e., wt = xπ(t) = xa:b. By leveraging the word boundary, we naturally break the long byte sequence into a list of sub-sequences, then we apply an Lw-layer Transformer encoder to encode byte tokens only in their sub-sequences:
Fπ(t) = Encoder(xπ(t), Lw),∀t ∈ [1 : T ], (8)
where Fπ(t) ∈ R|b−a|×d is hidden representation of byte tokens in the t-th word spanning over the sub-sequence xa:b. This allows byte tokens to effectively aggregate local information for each word token, which is useful for the model to distinguish identical byte tokens used in two different words. Note that the word-based self-attention in Eq. (8) can be efficiently implemented by pre-computing a block-wise attention mask matrix (Figure 1 (c)), ensuring that self-attenion is only performed among a byte span of a word in a Transformer layer. Finally we obtain the word-aware representation of the input byte sequence F by putting Fπ(t) in the word order, i.e., F = [Fπ(1), · · · ,Fπ(T )] ∈ RN×d, and feed F as input to the remaining (L − Lw) Transformer layers similar to Eq. (7).

heading: 4 Experimental Settings

heading: 4.1 Datasets
text: Multilingual Many-to-One Translation: We use the OPUS public data (Tiedemann, 2012) to construct a multilingual parallel corpus that has a fair mix of high-resource and low-resource languages. We train a multilingual translation model from seven source languages to English. Table 1 shows the statistics of the training data. We use the Flores-101 (Goyal et al., 2022) benchmark to evaluate the performance of our models.
Zero-shot Cross-lingual Translation: Following Neubig and Hu (2018), we use the same Ted Talk dataset that include four language pairs where each pair has a high-resource language (HRL) and a low-resource languages (LRL) written in the same script. Table 2 shows the statistics of the dataset.
Cross-domain Adaptation: In this task, we train all models on the WMT19 German-English dataset on the news domain, and directly evaluate on the test data used in Aharoni and Goldberg (2020) from three diverse domains (Koran, IT, Medical).

heading: 4.2 Models
text: To fully evaluate the efficacy of byte-based techniques, we consider models under settings where we learn dense embeddings for the input byte tokens (DENSE) as well as the embeddingless case where there are no learnt embeddings (ONE-HOT). Our main baseline for comparison is the vanilla byte-based model and the ONE-HOT model proposed in Shaham and Levy (2021). We also include
results of subword and character-based models for a holistic comparison.
Subword Model: We use BPE models trained using Sentencepiece3 as our subword model.
Char Model: We use character-based models with inputs and outputs being character sequences.
Byte Based Models: For each of these models, we consider both ONE-HOT variants where models do not have learnt embeddings and DENSE variants where we learn continuous dense embeddings.
• Byte: Similar to the vanilla byte-based model proposed by (Shaham and Levy, 2021), the inputs and outputs are UTF-8 byte tokens.
• Byte-nCF: We use a shallow Transformer encoder4 (Ls = 1) and four convolutional layers to fuse character-level information and learn a weighted n-gram representation (§3.1)
• Byte-WSF: We use a Lw-layer Transformer encoder5 with a word-based self-attention over byte tokens within word boundaries (§3.2).

heading: 4.3 Multilingual Translation
text: In this experiment, we evaluate the subword and byte-based models on many-to-one translation (xxeng) where xx refers to seven source languages listed in Table 1. We first clean the training data by removing sentences that are longer than 800 bytes in either the source or the target side, and then tokenize the sentences using the Moses tokenizer.6 Doing such preprocessing does not affect the diversity of the dataset in terms of length as less than 0.5% of the samples are discarded. The byte-based models do not have any preprocessing apart from 3https://github.com/google/sentencepiece 4Appendix C shows that Byte-nCF works best on {deu,khm}eng translations when fusing lower-layer representations. 5Appendix D shows that Lw = 4 empiricall works best. 6https://github.com/moses-smt/ mosesdecoder
the Moses tokenization and even whitespaces are included as valid tokens. For low-resource languages that share the same script as high-resource languages, we can reuse the same tokenizer for the high-resource language. For the subword-based model, we construct a shared vocabulary of 64K BPE tokens for all the source languages and an English vocabulary of 8K BPE tokens for this experiment. All models are trained for the same number of epochs on our OPUS train set, and evaluated on the Flores-101 test set.

heading: 4.4 Cross-lingual Transfer
text: This experiment evaluates how effective subword and byte-based methods are in transferring performance across languages that share similar language scripts. We train both the subword and byte-based models on parallel data in a high-resource language (HRL) for 50K steps, and evaluate them in a zeroshot manner on the corresponding low-resource language (LRL) without training on any LRL data. Table 2 shows the data statistics. We focus on xxeng translation where xx is either HRL or LRL.
In the case of subword models, this amounts to constructing a vocabulary (i.e., BPE tokenizer) based on only the HRL data and using that to tokenize the LRL data, while byte-based models use an universal tokenization scheme to tokenize both HRL and LRL data into UTF-8 byte tokens.
We also investigate a few-shot setting where the models pre-trained on the HRL data is further finetuned on a few parallel training samples in LRL. We examine the impact of different numbers (i.e., 1K, 2K, 3K, and 4K) of few-shot samples on the translation performance of these models. We finetune all models for 5K steps on the few-shot samples, and then evaluate them on the test set in LRL.

heading: 4.5 Cross-domain Adaptation
text: Having translation systems adapt to domains apart from the one it has been trained on is a good measure of how robust models are. In the experiment, we compare subword and byte-based models on how effectively they translate sentences from do-
mains that are not part of the training set. Similar to the cross-lingual transfer setting (§4.4), we train both subword and byte-based models on the source domain (News) and evaluate them in a zero-shot manner on three target domains (Koran, IT, Medical). Each model is trained on the source domain dataset for 50K steps, and then evaluated on each of the target domain test sets.

heading: 4.6 Hyperparameters
text: We use the Fairseq7 library as the codebase. To make a fair comparison, we strictly follow the architectural choice of Shaham and Levy (2021) and employ the vanilla transformer encoder-decoder architecture as our backbone for all experiments. For all models, we use a total of 6 Transformer layers for the encoder and 6 layers for the decoder with 8 attention heads, 512 hidden units and the feed-forward dimension of 2048. We use the Adam(Kingma and Ba, 2014) optimizer with an inverseq square root learning rate scheduler, and warm up 4K steps to reach a peak learning rate of 5e-4. We apply a weight decay of 1e-4 and a label smoothing of 0.1. We also train all models for an equal number of epochs in all the experiments.

heading: 4.7 Evaluation
text: For a fair, consistent evaluation, we follow Shaham and Levy (2021) in using Sacre-BLEU8 with 13a tokenizer for all language pairs using the raw text to compute the BLEU scores.

heading: 5 Results
text: In this section, we detail the results of the various experiments and discuss their implications.
7https://github.com/facebookresearch/ fairseq 8https://github.com/mjpost/sacrebleu

heading: 5.1 Multilingual Translation
text: Table 3 shows the BLEU scores on the test set of the FLORES-101 data for many-to-one translation. We further investigate the following questions.
Do we need dense embeddings? In line with the findings of (Shaham and Levy, 2021) that embeddingless models are competitive with subword models in bilingual settings, we find that they perform on par with their corresponding models that use dense embeddings with an average difference of less than 0.5 BLEU over seven languages. We find that for six out of the seven source languages under consideration, the byte-based models perform competitively with the subword models. However, subword models still hold the edge for extremely low-resource languages such as Lao-English translation with only 2.7K training data. Besides, as Lao’s written script is not shared with any of the other languages, we hypothesize that training bytebased multilingual models requires more training data in order to figure out the different fusion of byte tokens across languages while subword models with an explicit vocabulary for all languages do not have this requirement.
How effective is character/word fusion? Our proposed methods (Byte-nCF and Byte-WSF) that induce higher level semantic representations for bytes improve over vanilla byte-based models in both cases (ONE-HOT and DENSE models) on all language pairs with an average gain of up to 1.4 BLEU. Since sequence lengths tend to be extremely long when using byte sequences, aggregating information locally in the lower layers enables the model to quantitatively obtain higher scores than even subword-based models except in the extremely lowresource regime.
Which fusion works better? Comparing our two proposed variants, the Byte-nCF model per-
forms slightly better than the Byte-WSF model in the DENSE case, while both perform comparably in the ONE-HOT case. In particular, BytenCF performs better than Byte-WSF on relatively high-resource languages (e.g., German, Hindi) with more than 1M training data. Besides, both variants perform comparably on low-resource languages (e.g., Khmer, Lao) with large fertility scores.

heading: 5.2 Cross-lingual Transfer
text: The performance of byte-based and subword models on cross-lingual transfer is shown in Table 4. As Byte-WSF and Byte-nCF have shown comparable performances in Table 3, we only include the Byte-nCF variant in the comparison below.
Does universal tokenization work? When evaluating subword and byte-based models in a zeroshot setting (§4.4), byte-based models outperform subword baselines by a clear margin of up to 6.1 average BLEU over all languages. The gains compared to vanilla byte baseline is 1.2 BLEU for DENSE variant and 0.7 BLEU for ONE-HOT models. Our results indicate that even for languages written in the same script, a rigid subword schedule is infeasible for NMT models to perform an effective cross-lingual transfer. Particularly, we observe a significant increase in BLEU in the glg-eng and slk-eng translations when using byte tokens as inputs.
Does fusion help cross-lingual transfer? We find that using the Byte-nCF fusion variant leads to marginal improvement over the vanilla byte-based model with an average gain of up to 0.4 BLEU. It should be noted that most of these language pairs share the same script and hence the convolution fusion technique works very well. Investigating whether such fusion techniques work for languages that do not share the same script can be explored in future work.
Does the few-shot setting improve performance? Figure 2 shows the translation performance in terms of BLEU for the BPE and byte-based models in the few-shot setting. We find that as the number of training data in LRL increases, the performance of the byte-based models improves, and Byte-nCF consistently improves over the vanilla byte model. The BPE baseline suffers from the issue of having a high unknown token rate and cannot take full advantage of the additional training data.

heading: 5.3 Cross-Domain Adaptation
text: Table 5 shows the results of subword and bytebased models on zero-shot cross-domain adaptation. The first row indicates the BLEU scores on the in-domain test set (WMT19 News), and the other rows showcase the performance on out-of-domain test sets (Koran, IT, Medical).
Are byte-based models robust to domain shift? We find that the performance of both subword and byte-based models is susceptible to domain shifts. The BLEU scores on other domains are significantly lower for all variants. However, on comparison, we find that byte-based models are more robust than subword models against domain, yielding higher BLEU scores on all out-of-domain test sets.
Employing convolution fusion with the bytebased models improves performance over subwordbased models, especially in the IT and medical domains. The issue with cross-domain adaptation remains that each new domain consists of specific jargon and entities that are not captured in the source
domain. This inhibits the models from capturing the required semantic information to translate outof-domain sentences effectively.

heading: 6 Discussion and Analysis
text: Next, we further present a qualitative analysis of byte-based models and our proposed variants.
We use the compare-mt toolkit (Neubig et al., 2019) to holistically analyze how the outputs of these models differ and what aspects different models excel at. We compare the multilingual NMT models (§5.1) on the German-English translation as a sample language pair for the analysis. Specifically, we group all source words in the test sentences into buckets by the word fertility score (Figure 4a) and the word length in terms of characters (Figure 4b). Recall that the word fertility score measures how many subword units a word is broken into, and we use the BPE tokenizer used in Section 5.1. We evaluate byte-based models (i.e., Byte, Byte-WSF, Byte-nCF) using one-hot representations on each group in terms of source word translation accuracy.
As German is a high-resource language, most German words (45%) have a fertility score of 1 implying that they are not segmented by the BPE tokenizer, and all byte-based methods perform comparable on these words. We find that the BytenCF method performs better than the other two byte-based methods on oversegmented words (as indicated by the accuracy on words with fertility scores above 4). We also find that the Byte-nCF method outperforms other methods on translating long words (depicted by the accuracy on words with length greater than 15 characters). Comparing to the word-based (Byte-WSF) or sentence-level full self-attention (Byte), we hypothesize that this is a result of encoding a smaller sequence length when using the convolution fusion operation, reducing the pressure of byte-based models to capture too much information from a span of byte tokens.

heading: 7 Related Work
text: Subword Models Byte Pair Encoding (Sennrich et al., 2016), Wordpiece (Wu et al., 2016) and SentencepPiece (Kudo and Richardson, 2018) are widely-used subword tokenization schemes for NMT models, or perhaps most neural NLP models. However, the rigid tokenization scheme poses challenges in terms of oversegmenting low-resource languages and adapting a pre-trained model to new
languages or new domains of different corpora (Sun et al., 2020; Bostrom and Durrett, 2020; Provilkov et al., 2020; Kudo, 2018; Godey et al., 2022).
Character-level Models Applying neural models directly on character sequences has been extensively studied (Sutskever et al., 2011; Graves, 2013; Kalchbrenner et al., 2016; Zilly et al., 2017; Melis et al., 2018; Al-Rfou et al., 2019; Kim et al., 2016; Gao et al., 2020; Tay et al., 2022). Characteraware methods were mainly developed by the use of word boundaries and convolutions over characters (Kim et al., 2015; Jozefowicz et al., 2016; Peters et al., 2018; El Boukkouri et al., 2020; Ma et al., 2020). However, for machine translation, character-based NMT models (Lee et al., 2017; Cherry et al., 2018; Libovický et al., 2022) still suffer from a high computational overhead of encoding and decoding much longer sequences.
Tokenization-free Methods Recent attempts have focused on using Unicode or UTF-8 encoding scheme to remove pre-built subword tokenizers from preprocessing. These byte-based methods have achieved promising results in terms of accuracy and speed-up in several multilingual language understanding tasks (Clark et al., 2021; Tay et al., 2021; Xue et al., 2021) or bilingual translation tasks (Shaham and Levy, 2021), while their application to multilingual or cross-lingual/domain settings is still underexplored.

heading: 8 Conclusion
text: We propose a Local Byte Fusion (LOBEF) method with two variants—one employing convolutions on byte n-grams and the other utilizing word-based self-attention restricted to word boundaries. We show that these two fusion variants improve upon vanilla byte-based models indicating that neural machine translation models benefit from the explicit aggregation of local semantic information for characters or words at lower layers of neural networks. Our experiments show that both ONE-HOT and DENSE versions of byte-based models perform competitively on multilingual machine translation and even beat subword baselines on multiple language pairs. We also conduct an investigation of the effectiveness of byte-based techniques in both zero-/few-shot cross-lingual transfer and domain adaptation settings, and find that they outperform subword models by a large margin.

heading: 9 Limitations
text: Despite achieving high translation performance on various language pairs, LOBEF has some limitations, coming from the nature of processing UTF-8 byte sequences.
Speed: As shown in Table 8 in the appendix, the inference times for byte-based models are higher when compared to subword-based models. It is also worth noting that we use the same amounts of model parameters for a total of 6 Transformer encoder layers and 6 Transformer decoder layers for all models in comparison. As shown in Table 7, byte-based models can effectively reduce the amounts of parameters for the embedding layers comparing to the subword-based models, leading to faster training time as shown in Table 8. However, as indicated by Xue et al. (2021), by adding more encoder layers, we can construct byte-based models with comparable amounts of parameters as subword-based models, and these larger byte-based models still require much longer time for training than subword-based models.
Extremely Low-resource Languages: The performance of byte-based models on extremely lowresource languages (e.g., 2.7K training data for Lao-English) is still lower than subword models especially in the multilingual setting. We suspect that byte-based methods require a relatively larger number of training data in order to aggregate information from a combination of byte tokens, comparing to subword-based models that explicitly maintain a subword vocabulary.
Extra Preprocessing: The Byte-WSF model requires an extra preprocessing step that precomputes the attention mask corresponding to the words in each sentence. This adds a slight overhead before training, while training the Byte-WSF model is as fast as the Byte model, as both model use the same Transformer architecture. However,
for languages (e.g., Chinese) that do not have whitespaces to indicate the word boundary, we may rely on an off-the-shell word segmentation tool to preprocess the text.

heading: A Datasets used for Multilingual Training

heading: B Computational Efficiency
text: For comparing the byte based and subword models in terms of the number of parameters, training and inference times we consider the transformer base architecture. For the subword baseline, we consider a source vocabulary of 32k and 8k for the target vocabulary (English).
Parameters When comparing the number of parameters in subword and byte based models, we find that byte based models have far fewer parameters (~30% fewer) as compared to the subword baselines. We highlight the differences in Table 7.
Training and Inference times For comparing the training times, we train the models on the WMT19 de-en data for 5k steps. We use a warmup of 1k steps to eliminate any hardware discrepancies like GPU9 cold starts. Since the byte based models are smaller than comparable subword models, they are 20% faster to train. The inference times are based on evaluating the model on the validation set across all batches10. We find that byte based models are significantly slower than subword models for inference. The byte sequences are significantly longer than subword sequences and thus the decoding time takes a hit. Table 8 shows the training and inference times for the subword and byte baesd models.
It should be noted that while the training time is shorter for byte-based approaches when comparing the same number of gradient steps, when we consider the training time for the same number of epochs, we do not observe faster training performance. Since byte sequences are much longer than subword sequences, training them for the same number of epochs involves using a longer number of training steps which makes their training times comparable. 9All numbers are obtained using a single RTX 3090 GPU using a batch size of 7k tokens and 8 gradient accumulation steps for training. 10beam size of 3 and batch size of 256
Computing Infrastructure All models are trained on a Linux server with 4 RTX 3090 GPUs and 16 CPU cores. On average, training all models on 2 GPUs for 200K steps can be finished within 24 hours. After training, we pick the best checkpoints based on the performance on the development set.
C Number of Shallow Encoding Layers for Byte-nCF

heading: D Number of Word-based Self-Attention Layers for Byte-WSF

heading: E Byte-BPE Baseline
text: There are some works exploring the use of BPE vocabulary on byte tokens (Wang et al., 2019a) to get the best of both worlds - i.e. we would not have the out-of-vocabulary issue since every character can be represented as one of the 256 byte tokens and we also make use of the advantages of subword tokenization scheme to reduce the sequence length and decoding time. For a more reasonable comparison with similar-sized byte-based models, we strictly follow the settings of Wang et al. (2019b), using BBPE models and setting vocab size to 2K or 4K. From our results below, we find that our proposed methods for byte fusion (using 256 vocab size) are slightly better than BBPE with 4K vocab size, with an avg. gain of up to 0.8 BLEU.
Note that BBPE may fall back to using single bytes when dealing with new byte combinations in a new language or domain. We also run a cross-lingual transfer experiment by training the BBPE(4k) model on tur-eng and evaluating it on aze-eng in a zero-shot manner. We find that it gets a BLEU of 7.4, which is better than vanilla byte models (6.9) but worse than our proposed Byte-nCF (7.8 BLEU). This suggests that even though there is a byte fall-back in such models, a certain fraction of BPE tokens used in high-resource languages may not be used in the low-resource language, and the model still has to implicitly fuse the new byte tokens for low-resource language, similar to vanilla byte baseline.
This baseline is more comparable to BPE and is not the main baseline for our consideration since we are focused on improving over the vanilla byte-based methods.
IT Medical 0
0.05
0.1
0.15
0.2
0.25
0.3
0.35 BPE Byte Byte-nCF
Domains
A cc ur ac
y
(a) Word accuracy for seen words during training
IT Medical 0
0.05
0.1
0.15
0.2 BPE Byte Byte-nCF
Domains
A cc ur ac
y
(b) Word accuracy on unseen words during training
Figure 4: Translation performance (indicated by Word Accuracy) grouped by whether the words were seen or unseen during training on different domains for ONE-HOT Byte models on deu-eng.
s F Seen vs Unseen words - Domain Adaptation
To compare how well the models generalize across domains, we compute the word accuracy score based on whether the source words were observed or unobserved during the training stage. Since all the models are trained on the WMT News domain and evaluated on the IT and Medical domains, analyzing the word accuracy on unseen words reveals where the performance gain stems from. We see that the Byte-nCF model has much higher word accuracies on the unseen words when compared with the Byte and BPE models
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Left blank.
3 A2. Did you discuss any potential risks of your work? Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Investigating whether such fusion techniques work for languages that do not share the same script can be explored in future work. Does the few-shot setting improve performance? Figure 2 shows the translation performance in terms of BLEU for the BPE and byte-based models in the few-shot setting. We find that as the number of training data in LRL increases, the performance of the byte-based models improves, and Byte-nCF consistently improves over the vanilla byte model. The BPE baseline suffers from the issue of having a high unknown token rate and cannot take full advantage of the additional training data