Deterministic Reversible Data Augmentation for Neural Machine Translation (2024)

Abstract: Data augmentation is an effective way to diversify corpora in machine translation, but previous methods may introduce semantic inconsistency between original and augmented data because of irreversible operations and random subword sampling procedures. To generate both symbolically diverse and semantically consistent augmentation data, we propose Deterministic Reversible Data Augmentation (DRDA), a simple but effective data augmentation method for neural machine translation. DRDA adopts deterministic segmentations and reversible operations to generate multi-granularity subword representations and pulls them closer together with multi-view techniques. With no extra corpora or model changes required, DRDA outperforms strong baselines on several translation tasks with a clear margin (up to 4.3 BLEU gain over Transformer) and exhibits good robustness in noisy, low-resource, and cross-domain datasets. 1

heading: 1 Introduction
text: Recent neural machine translation (NMT) models have led to dramatic improvements in translation quality. However, the powerful learning and memorizing ability of these models also leads to poor generalization and vulnerability to small perturbations like misspelling and paraphrasing (Belinkov and Bisk, 2017; Cheng et al., 2020).
A common solution to perturbation vulnerability is data augmentation (Sennrich et al., 2016b; Cheng et al., 2016), which is to create massive virtual training data with diverse symbolic representations under the premise of ensuring semantic consistency (Cheng et al., 2019, 2020). Symbolic diversity emphasizes that original and augmented data should differ significantly in token sequences, and semantic consistency requires that the two should be semantically similar. Previous data
1The relevant code is available at https://github. com/BITHLP/DRDA.
augmentation methods employ irreversible substitutions, like direct dropping or replacing discrete tokens to generate diverse data (Figure 1 A). Despite being able to improve data diversity, these augmentation operations are not reversible, and will inevitably introduce semantic loss to original texts, thus compromising the semantic consistency between original and augmented data.
Yet another way to generate diverse augmentation data without employing irreversible operations is subword regularization (Kudo, 2018; Provilkov et al., 2020). Subword regularization adopts random segmentations to sample subwords probabilistically thus generating diverse data. These methods are reversible because of the inherent reversibility of segmentations. However, due to the random sampling procedure of segmentation, they may adopt inappropriate subword segmentations (e.g., "sup erm ark et" in Figure 1 B). These sub-optimal segmentations may result in semantic perturbations and do damage to semantic consistency.
To summarize, previous methods have difficulty in completely retaining the semantics from corruption when diversifying the texts because of irreversible augmentation operations and probabilistic subword sampling.
To generate symbolically diverse and semantically consistent data, we propose Deterministic Reversible Data Augmentation (DRDA), a simple but effective augmentation approach. DRDA augments source sentences with their token representations in different granularities as shown in Figure 1 C. These representations are symbolically diverse, but also syntactically correct and semantically complete thanks to the reversible and deterministic segmentations in the multi-granularity segmentation process. To make full use of the semantic identity among all multi-granularity representations of one sentence, we also leverage the multi-view techniques in training to pull these representations closer together.
8075
We conduct extensive experiments of different languages and scales and find that DRDA gains consistent improvements over strong baselines with clear margins. To further understand the factors that make DRDA work, we conduct insightful analyses of the effects DRDA imposed on semantic consistency, subword frequency, and subword semantic composition. We combine the empirical and theoretical verification of the consistency and offer a subword-level explanation of the mechanism of multi-granularity segmentations and multi-view techniques.
Our contributions are summarized as follows:
• We propose DRDA that exclusively employs deterministic reversible operations to generate diverse augmentation data without introducing semantic noise.
• We conduct extensive experiments and verify the high effectiveness of DRDA.
• To investigate the factors that make DRDA work, we combine empirical and theoretical analyses and offer insightful explanations.

heading: 2 Related Work
text: Augmentation methods Besides continuous approaches (Wei et al., 2022), data augmentation can be categorized into back-translation like methods (Sennrich et al., 2016b; Edunov et al., 2018; Nguyen et al., 2020) and token substitution methods. DRDA is an instance of the latter category.
Several substitution methods uniformly select a word or token in a sentence and perform deletion or substitution (Zhang et al., 2020; Shen et al., 2020; Wang et al., 2018b; Norouzi et al., 2016; Gao et al., 2022). Cheng et al. (2019, 2020) constrained
the substitution of a word in a small subset of synonyms, thus improving the semantics consistency. Kambhatla et al. (2022b) viewed the original corpus as plain text and applies a rotation encryption as data augmentation. Unlike previous methods, introducing multi-granularity takes advantage of the reversible nature of segmentation and causes no semantic loss.
Subword regularization The de-facto subword method, BPE (Sennrich et al., 2016c), still suffers from sub-optimality (Bostrom and Durrett, 2020). To overcome this sub-optimality, several subword regularization approaches are proposed. Kudo (2018) and Provilkov et al. (2020) presented subword regularization by modelling segmentation ambiguity. Wang et al. (2021) integrated BPE and BPE-Drop by enforcing the consistency using multi-view subword regularization, Wu et al. (2020) and Kambhatla et al. (2022a) combined BPE in SentencePiece and subword-nmt together to obtain regularization effects. DRDA is distinct from all the random sampling segmentation methods, as the augmentation data is generated deterministically. The determinism helps alleviate less reasonable segmentation, while achieving regularization effects as well.
In addition, other researches put efforts into taking advantage of multi-granularity representations, which can also be viewed as a subword regularization. Li et al. (2020) and Gao et al. (2020) adopted word lattice and convolutions of different kernel sizes respectively, Chen et al. (2018) and Li et al. (2022) combined levels of representation scales, Hao et al. (2019) modified self-attention module to introduce phrase modeling. Unlike these methods, DRDA requires no modification to model architectures and can be applied to universal tasks.

heading: 3 Background: Subword Segmentation
text: Subword segmentation models the probability of token sequence x = x1, x2, ..., xm given a source sentence s. Previous deterministic subword segmentations choose the most probable sample:
x∗ = argmax x Pseg(x|s; p)
= argmax x∈Vp
Pseg(x|s), (1)
where p is the size of the vocabulary (a set of subword candidates), and each token xi (i ∈ {1, 2, ...,m}) is selected from vocabulary Vp. For example, Byte Pair Encoding (BPE) assigns P (x̂|s; p) = 1 when x̂ is obtained from the greedy merge process (Sennrich et al., 2016c).
To generate different segmentations for one word, subword regularization methods draw a segmentation from the segmentation distribution probabilistically:
x ∼ Pseg(x|s; p). (2) For example, Kudo (2018) makes use of a unigram language model to sample segmentations on, and Provilkov et al. (2020) randomly interrupts the BPE merging process to generate multiple segmentations.

heading: 4 Deterministic Reversible Data Augmentation
text: Previous data augmentation and subword regularization approaches take irreversible operation (like discrete token substitution) and probabilistic segmentation sampling, which may introduce semantic loss or inappropriate subwords, thus affecting the semantic consistency. Our objective is to ensure the semantic consistency between original and augmented data when generating diverse data.
We propose DRDA to generate augmentation data without introducing semantic perturbations. DRDA augments original data with multigranularity segmentations, and pulls representations of one sentence closer with multi-view learning. Furthermore, we propose a dynamic selection technique to automatically choose an appropriate granularity in inference.

heading: 4.1 Multi-Granularity Segmentations
text: DRDA constructs symbolically diverse and semantically consistent augmentation data with multigranularity segmentations. The point is that multigranularity subword segmentation is a reversible
process that completely retains semantic information, and is a deterministic process that always chooses the most probable and appropriate subword segmentation policy.
Formally, given a prime vocabulary size p and a set of augmented vocabulary sizes {qi}ki=1, for a source-target translation pair sample (s, t), a prime source sequence xpri, a target sequence y and a set of augmented source sequences {xaugi}ki=1 can be generated:
xpri = argmax x∈Vp P (x|s), (3)
xaugi = argmax x∈Vqi P (x|s), (4)
y = argmax y′∈Vp
P (y′|t). (5)
Figure 2 depicts the model architecture and training loss on a English→Germany sample. Given p = 12000, q1 = 1000, and q2 = 6000, an English sentence is segmented with different vocabularies, generating three token sequences with different granularities.
Note that according to the greedy property of BPE, a short vocabulary is a prefix of a long vocabulary, as long as they are obtained from the same corpus. As a result, introducing different granularities with BPE will not lead to a larger vocabulary, thus avoiding an increase in parameter size. An example is shown in Figure 2, where three embedding matrices E12000, E6000 and E1000 are overlapped, and a smaller embedding is a prefix of a larger embedding.

heading: 4.2 Multi-view Learning
text: Moreover, to make the translation model learn from different segmentation granularities, we utilize the multi-view learning loss function (Wang et al., 2021; Kambhatla et al., 2022b) and pull different representations closer together:
L = LNLL(P (y|xpri; θ))︸ ︷︷ ︸ prime source loss
+ 1
k
k∑
i=1 LNLL(P (y|xaugi ; θ)) ︸ ︷︷ ︸
augmented source loss
+ α
k
k∑
i=1 Ldist(P (y|xpri; θ), P (y|xaugi ; θ)) ︸ ︷︷ ︸
agreement loss
,
(6) where LNLL is the negative likelihood loss in machine translation, Ldist is the symmetric KullbackLeibler divergence (Kambhatla et al., 2022b).
The first two terms of Equation 6 (prime source loss and augmented source loss) compute the translation loss for source and augmented sentences respectively, and the third term (agreement loss) pulls the prediction distributions of different source inputs together.
As shown in Figure 2, output probability distributions for all granularities are used to compute the loss, where the blue segmented lines refer to the agreement loss between different granularities, and green dotted lines refer to the negative likelihood loss between the prediction and the target.

heading: 4.3 Dynamic Selection of Granularity in Inference
text: DRDA employs multiple segmentations in different granularities, so the selection of the granularity used in inference becomes a concern. To automatically choose a suitable vocabulary size when inferring, we also propose a simplified but granularityfocused version of n-best decoding (Kudo, 2018) to dynamically select the segmentation granularity in inferring step.
Given the set of all prime and augmented vocabulary sizes {p, q1, q2, · · · , qk} and an input sentence s, a series of (x,y) pairs can be generated, where each (x,y) pair represents a source-target token sequence pair in a certain granularity.
The estimated most probable segmentation and translation pair corresponds to the (x,y) pair that maximizes the following score:
score(x,y) = logP (y|x)/|y|, (7)
where |y| is the length of y.

heading: 5 Experiments
text: We evaluate DRDA with translation tasks in different language pairs and translation directions to show its universal property regardless of language features. We also conduct experiments on extremely low resources and noisy scenarios to show the robustness of DRDA.2

heading: 5.1 Experimental Setup
text: 2Further setup details about dataset split, preprocessing, models, and evaluation are listed in Appendix A.
Datasets and preprocessing Our experiments are conducted on different datasets, as detailed in Table 1. We experiment on a low resource setting with IWSLT datasets, including IWSLT14 En↔De, En↔Es, and IWSLT17 En↔Zh, En↔Fr. We use larger WMT14 En↔De as a high-resource scenario dataset. The performance in extremely low resource scenarios is explored with the TED En↔Sk dataset. Following previous work (Vaswani et al., 2017), we lowercase words in IWSLT En↔De, while keeping other datasets cased.3
Models We build models on top of Transformer (Vaswani et al., 2017) with Fairseq toolkit (Ott et al., 2019). We use a Base Transformer model transformer_wmt_en_de for WMT, and transformer_iwslt_de_en for others.

heading: Hyperparameters in training and inferring
text: We use sentencepiece (Kudo and Richardson, 2018) to perform tokenization and BPE segmentation. The BPE encoding model is learned jointly on the source and target sides except for IWSLT En↔Zh. Unless otherwise stated, we use two vocabulary tables (on prime vocabulary and one augmented vocabulary), and their vocabulary sizes follow Table 3. Detailed analysis of the vocabulary sizes and the number of augmented vocabularies will be shown in Section 6.1. The weight of agreement loss α is set to 5 unless otherwise stated.
Evaluation We evaluate the performance of NMT systems using BLEU. To compare with
3En, De, Fr, Zh, Es, Sk stand for English, German, French, Chinese, Spanish, and Slovak respectively.
previous work (Vaswani et al., 2017; Kambhatla et al., 2022b), we apply multi-bleu with multi_bleu.perl 4 for IWSLT En↔De, WMT En→De, and TED En↔Sk. For WMT En→De dataset, we additionally apply compound splitting5. All other datasets are evaluated with SacreBLEU6.

heading: 5.2 Main Result
text: We present the results of DRDA on IWSLT and WMT translation tasks in Table 2. We can see that DRDA consistently outperforms the Transformer with a clear margin on all translation tasks. Moreover, models inferred with the dynamic granularity selection obtain a modest improvement in DRDA.

heading: Model En→De De→En
text: Comparison between DRDA and other data augmentation and subword regularization methods on IWSLT are shown in Table 4. We use a range of
4mosesdecoder/scripts/generic/multi-bleu.perl 5tensorflow/tensor2tensor/utils/get_ende_bleu.sh 6SacreBLEU signature: nrefs:1|case:mixed|
eff:no|tok:13a|smooth:exp|version:2.2.0
augmentation and regularization methods for comparison. The augmentation methods include WordDrop (Zhang et al., 2020; Sennrich et al., 2016a), SwitchOut (Wang et al., 2018b), RAML (Norouzi et al., 2016) and Data Diversification (Nguyen et al., 2020). The subword regularization methods include BPE-Drop (Provilkov et al., 2020) and Subword Regularization (Kudo, 2018). We also compare our method with others that adopt multi-view learning techniques, including R-Drop (Wu et al., 2021), MVR (Wang et al., 2021), and CipherDAug (Kambhatla et al., 2022b). DRDA yields greater improvement compared to others.

heading: 5.3 Extremely Low Resource Setting
text: TED En↔Sk task is challenging because of its extremely low resources (only 61k training sentence pairs). Several techniques have been adopted to improve the performance in low-resource NMT tasks like this, including data augmentation, multilingual translation, and transfer learning (Ranathunga et al., 2021). Neubig and Hu (2018) firstly propose similar language regularization to mix low-resource language with a lexically related high-resource language, combining transfer learning and multilingual translation. Several works continue to extend SRL and achieve high translation quality (Xia et al., 2019; Ko et al., 2021; Wang et al., 2018a).
On this task, DRDA yields stronger improvements over baseline Transformer than other techniques with no requirement for external high resource languages, as shown in Table 5.

heading: 5.4 Robustness to Perturbations
text: We validate the robustness of DRDA on two noisy datasets. The first one is IWSLT De→En test set with synthetic perturbations. The perturbations are synthesized by traversing every character excluding
space and punctuation in source sentences, and applying one of the operations with probability 0.01: (1) remove the character, (2) add a random character following the character, and (3) substitute the character with a random one. The second dataset is himl test set7, which contains health information and scientific summaries and differs considerably from the IWSLT training set. Cross-domain datasets have different subword distributions, and the difference can be viewed as a natural noise. The results of the noisy test sets are shown in Table 6.
Along with these results, consistent improvement over Transformer and R-Drop is obtained by DRDA on both synthetically noisy and crossdomain datasets. DRDA significantly outperforms subword sampling methods (BPE-Drop and subword regularization) on natural noise datasets, but only obtains similar results with synthetic noise. We will discuss the reason in Section 6.2.

heading: 6 Analysis
text: In this section, we conduct analysis experiments to answer the following research questions (RQs) respectively:
• RQ1 (ablation studies): How do the applied techniques and components affect model performance?
• RQ2: Does our approach really keeps semantic consistency between original and augmented data?
• RQ3: How does multi-granularity segmentation improve subword representations?
• RQ4: Why does multi-view learning help improve NMT models?
7https://www.himl.eu/test-sets

heading: 6.1 RQ1: Ablations
text: Choice of vocabulary sizes Here, we investigate the effects of pre-defined vocabulary sizes. As is mentioned in Section 5.1, we adopt one prime vocabulary and one augmented vocabulary. To find the optimal vocabulary sizes, we test {10k, 7k, 5k, 3k, 1k} for augmented vocabulary size when the prime size is 10k. Figure 3 verifies that, when the augmented vocabulary size is around 5k, the NMT model obtains the highest BLEU. The intuition is a huge difference in prime and augmented vocabulary sizes may corrupt the subword semantics, while a tiny difference may reduce the symbolic difference. A general recommendation in choosing vocabulary sizes is to use a proven suitable size for the prime vocabulary and set the augmented size to half the size of the prime vocabulary.
Weight of agreement loss As is shown in Figure 3, we find that agreement loss weight α significantly affects the performance of our method. Models obtain the highest BLEU score when α = 5, and increasing or decreasing α causes a score drop up to 2 BLEU on the valid set. The model without agreement loss (i.e., α = 0) still outperforms vanilla Transformer, validating the important role multi-granularity segmentation plays in DRDA.
Number of augmented vocabularies Table 7 shows the effects of adding an extra augmented vocabulary with a prime vocabulary size of 12k on the valid set. When combined with two augmented vocabularies, the BLEU scores have a smaller deviation than combined with one. We can summarize that adding extra augmented vocabularies helps get a steady, comparable, and maybe slightly better result in the cost of an increase in training time.

heading: 6.2 RQ2: Semantic Consistency
text: Theoretical discussion In this section, we discuss what is semantic consistency, and give a theoretical analysis about why DRDA is more semantically consistent.
It is clear that previous data augmentation methods that adopt irreversible operations result in semantic loss, which will inevitably do damage to the consistency between original and augmented data. DRDA is superior to these methods in terms of preservation of the original meanings, because it is based on reversible segmentation to generate diversity.
However, it is more challenging to prove that subword regularization methods (Kudo, 2018; Provilkov et al., 2020), which are also based on reversible segmentation, lead to greater inconsistency than DRDA. To show the superiority of DRDA in consistency over subword regularization, we review the difference of the two in sampling segmentation:
xiDRDA = argmax x Pseg(x|s; pi), (8)
xSR ∼ Pseg(x|s; p), (9) where xiDRDA is a representation in certain granularity of source sentence s in DRDA, xSR is the representation in subword regularization, pi and p are vocabulary sizes. argmax
x Pseg(x|s; p) can be interpreted as the
difficulty of segmenting s with a certain vocabulary size p. We can assume that the difficulty of segmenting a sentence is an inherent property of sentences, independent of vocabulary sizes:
argmax x Pseg(x|s) = argmax x Pseg(x|s; p), (10) where p ∈ N is any pre-defined vocabulary size. Then, because of the deterministic argmax operation in DRDA and the random sampling operation
in subword regularization, the following inequality holds:
Pseg(xDRDA|s) ≥ Pseg(xSR|s). (11)
Equation 11 validates that our approaches generates more appropriate segmentations of a same sentence that other subword regularization methods. As a result, although both DRDA and subword regularization are reversible, DRDA is semantically more consistent because of the segmentation appropriateness.
Empirical analyses To give an empirical insight of the semantical consistency, we analyze the nearest neighbors of subwords of different models (shown in Table 8). We can find that vanilla Transformer and DRDA both exhibit semantics-based neighbors, where the embeddings of synonyms are similar. However, embeddings obtained in BPE-Drop tend to have high similarity with those they share a common sequence. Although this tendency can effectively alleviate vulnerability to misspelling, which explains the superiority subword regularization shows in synthetic noisy data in Section 5.4, it may introduce semantic error as well (treat "_go" and "_god" as synonyms for "_good" in Table 8 for example), causing inaccuracy in machine translation.
The observation above indicates that DRDA introduces little semantic noise to augmentation data, and exhibits better semantic consistency.

heading: 6.3 RQ3: Effects on Subword Frequency
text: Here, we show that the mechanism of multigranularity segmentation can be attributed to the increase in frequency of infrequent tokens.
NMT models with larger vocabulary sizes have larger atomic translation units, i.e., more coarsegrained subwords, so that they can better memorize one-to-many or many-to-one mappings and
resolve translation ambiguity (Koehn, 2009). However, fine-grained subwords may suffer from a frequency drop when the vocabulary size grows. Figure 4 shows that most occurrences of "_nerv" are absorbed by "_nervous" when the vocabulary grows, making it more difficult for the NMT model to obtain a precise representation of other inflection forms like "_nervy", "_nervier" and "_nervine". More generally, the frequency drop is common on IWSLT En→De (results on more datasets are shown in Appendix C), where about 50% of subwords appeared in 5k vocabulary suffer from a frequency drop when the vocabulary grows to 10k, as Figure 4 shows.
In DRDA, by taking both small and large vocabulary sizes simultaneously, infrequent tokens occur more frequently so that subwords like "_nerv" can be trained in adequate contexts as well.

heading: 6.4 RQ4: Multi-view Techniques and Subword Semantic Composition
text: Multi-view learning pulls representations in different granularities together. To investigate the effects of multi-view techniques, we propose a task to find out how the coarse-grained and fine-grained representations of the same word are drawn closer.
The process is illustrated with an example in Figure 5, and the formal definition of the task is shown
in Appendix B. We take a coarse-grained subword ("_background") and its corresponding finegrained subword sequence ("_back", "ground"), then compute the cosine similarity between the former embedding and the sum of the latter embeddings. The similarity indicates the extent to which the fine-grained and coarse-grained representations are brought closer together.
We enumerate all the coarse-grained and finegrained representation pairs, and average all their cosine similarity scores. The results are shown in Table 9. As expected, DRDA with proper agreement loss (α = 5) obtains a higher average similarity than other data augmentation approaches.
Computing the similarities between representations in multiple granularities is a subword level composition (SSC) tasks (Mitchell and Lapata, 2008, 2009; Turney, 2014). We can conclude that multi-view techniques help DRDA models improve the SSC understanding, thus obtaining better robustness to perturbations (Provilkov et al., 2020).

heading: 7 Conclusion
text: In this paper, we identify the semantic inconsistency caused by irreversible operations or probabilistic segmentations, and propose a deterministic reversible data augmentation consisting of multigranularity segmentation and multi-view learning to ensure the consistency when generating diverse data. Experiments demonstrate the superiority of our proposed DRDA over previous data augmentation and subword regularization in terms of translation accuracy and robustness. We also offer a combination of empirical and theoretical verification of semantic consistency, and insightful analyses about multi-granularity and multi-view techniques.

heading: Limitations
text: High resource scenarios As other data augmentation techniques, our proposed DRDA appears to be less effective in high-resource scenarios (up to 1.75 BLEU gain in WMT, and 2.69 in IWSLT) than in low resource scenarios (up to 4.37 BLEU gain in TED). The analysis in Section 6.3 offers one explanation to this phenomenon that, the frequency drop becomes less sharp when the data size grows, thus resulting in lower effectiveness of data augmentation. Considering this phenomenon, a better application approach of data augmentation on high-resource scenarios can be designed, by locating the rare subwords of a specific domain in a model trained on large general corpus and continuing training with the augmentation data. We leave this investigation as a direction for future research.
Application scope As a foundation process in NLP, segmentation is applied in various tasks, including language modeling, named entity recognition, and numerous others. Additionally, vision tasks like image translation can also benefit from segmentation (Tian et al., 2023). As a result, segmentation based data augmentation techniques including DRDA can be applied to a wide range of tasks. One limitation of this study is its exclusive application of DRDA to machine translation, which restricts the ability to validate and compare its effectiveness across other tasks.

heading: Acknowledgements
text: We’d like to thank all the anonymous reviewers for their diligent efforts in helping us improve this work. This work is supported by the National Natural Science Foundation of China (Grant No. U21B2009) and Beijing Institute of Technology Science and Technology Innovation Plan (Grant No. 23CX13027).

heading: A.1 Datasets and Preprocessing
text: We perform minimum preprocessing and cleaning steps to raw data.
• For IWSLT En↔De, following previous works, data is obtained with fairseq scripts8, which performs clean-corpus-n 9 with ratio = 1.5, min = 1 and max = 175. • For other IWSLT datasets, we extract titles, descriptions and main texts for training, and main texts only for validating and testing. There is no extra cleanup operation performed. IWSLT14 En↔Es dataset concatenates dev2010, tst2010, tst2011 and tst2012 as development set, uses tst2015 as test set. IWSLT17 En↔Fr and En↔Zh datasets concatenate dev2010, tst2010, tst2011, tst2012, tst2013, tst2014 and tst2015 as development set, use tst2017 as test set. • We use t2t-datagen 10 script to generate WMT data, and performs clean-corpus-n with min = 1 and
8fairseq/example/translation//prepare_iwslt14.sh 9mosesdecoder/scripts/training/clean-corpus-n.perl
10tensorflow/tensor2tensor/bin/t2t-datagen
max = 80, removing about 1% training sentence pairs. Following previous works, we validate on newstest2013 and test on newstest2014. • The TED datasets are obtained using scripts from the official repository (Qi et al., 2018). We additionally remove the encoder language token "__sk__" to accommodate bilingual NMT.

heading: A.2 Models and Hyperparameters
text: Smaller datasets are trained with model transformer_iwslt_de_en, and WMT dataset is trained with model transformer_wmt_en_de. The corresponding config is shown in Table 10.
Note that DRDA, R-Drop, CipherDAug and some other approaches may double the input texts, but we constrain the tokens number forwarded to the model in a batch according to the "words per batch" hyperparameter, which means the numbers of sentences in a batch of these approaches are rough halved.

heading: A.3 Computational Cost
text: Total training duration and GPU used in DRDA experiments are listed in Table 11.

heading: A.4 Baseline Implementation
text: We reimplement those models with high relevance with our method, including vanilla Transformer, BPE-Drop, subword regularization, R-Drop, MVR and CipherDAaug. These models except for Transformer use either segmentation-related techniques or multi-view techniques. Important details of our implementation are listed below:
• BPE-Drop and subword regularization are implemented using sentencepiece. In encoding, we set α = 0.1 and α = 0.2 for BPEDrop and subword regularization respectively, and nbest_size = −1 for both. Results of subword regularization are obtained without n-best decoding (Kudo, 2018). • We use the task and loss module from the official open-source repository 11 to implement R-Drop. Weight α is set to 5. • In MVR implementation, we adopt the same subword regularization hyper-parameters to BPE-Drop, and the agreement loss weight is set to be 5. • CipherDAaug models are reimplemented on top of the official open-source code12. Following their instructions, we adopt 2 keys, and set agreement loss weight β = 5.
For the traditional data augmentation methods ( WordDrop, SwitchOut, RAML, DataDiverse) with which DRDA shares a relatively low similarity, results are cited from Kambhatla et al. (2022b). We share exactly the same model architecture and hyperparameters with Kambhatla et al. (2022b), and we successfully reimplemented their main model with similar results, so we find it reliable to cite from.
We report the performance of LRL+HRL from the corresponding literature (Xia et al., 2019).
11https://github.com/dropreg/R-Drop 12https://github.com/protonish/cipherdaug-nmt

heading: B Process of Subword Semantic Composition Task
text: Let a ◦ b be a compound token concatenated by a and b, with their corresponding embedding ea◦b, ea and eb, the SSC understanding ability is scored by the similarity between ea◦b and ea + eb:
SIM(ea◦b, ea + eb) = ea◦b · (ea + eb)
∥ea◦b∥ · ∥ea + eb∥ . (12)
To numerically evaluate the superiority of a model in understanding SSC, we average semantic composition similarities of all subwords except characters and special tokens (such as <unk>):
SIM = 1 |Ṽ | ∑ a,b,a◦b∈V SIM(ea◦b, ea + eb), (13)
where Ṽ is a set of all subwords except characters and special tokens, and V is a set of all subwords.
It should be noted that the models listed in Section 6.4 share the same V , so that comparing the scores completely makes sense.

heading: C More Studies

heading: C.1 Subword Nearest Neighbors
text: Following Provilkov et al. (2020), we study the closest neighbors of word embedding learned in BPE-Drop and DRDA. Several examples are shown in Table 12.
We can find that in the morphology of words, BPE-Drop tends to bring two subwords sharing a
common sequence together ("_good" and "ood" for example), while DRDA has no such behavior. On one hand, the tendency to pull similarly spelled words closer can effectively help NMT model overcome the perturbation of misspelling, as shown in previous experiments. On the other, it can introduce unreasonable noise as well, since similarly spelled subwords are not necessarily semantically related words ("_good" and "_go" for example).

heading: C.2 Effects of Granularity Selection
text: Our experiments have shown that the dynamic selection of segmentation granularity yields a modest improvement in BLUE scores. Here, we investigate the mechanism and potential of this method.
We define an oracle granularity selection model, whose translation result corresponds to the one with the highest sentence-level BLEU score among the results generated by source sequences with different granularities. The results of models with 5k augmented size and 10k prime size on IWSLT translation tasks are shown in Table 13.
It can be summarized from the results that the selection of input granularities has considerable potential (up to 1.7 BLEU) in improving the translation, and our approach obtains an improvement of up to 0.24 BLEU. In the future, a better re-ranking approach can be adopted to build a selection model closer to the oracle model.

heading: C.3 Frequency Drops
text: More examples of frequency drop on WMT, IWSLT and TED are shown in Figure 6. Among these results, all datasets suffer from a similar frequency drop regardless of their language directions and sizes. The vocabulary size grows from 16k to 32k for WMT, from 5k to 10k for IWSLT and from 4k to 8k for TED.

heading: D Discussion on Potential Application in Language Modeling
text: As a foundation process in NLP, segmentation is applied in various tasks. As a result, segmentation based data augmentation techniques including DRDA can also be applied to a wide range of tasks including language modeling. Considering the significant upsurge in the development of large language models (LLMs) (Brown et al., 2020; Achiam et al., 2023) and their relevance to machine translation (Hendy et al., 2023), we discuss the potential application of DRDA in language modeling.
In our preliminary view, DRDA may be beneficial in language modeling on training or fine-tuning steps. Specifically, consider the scenario where a language model is trained using the IWSLT dataset. In such instances, the model will encounter the issue described in Section 6.3, where the limited contextual information can hinder its ability to accurately capture the diverse inflectional forms of a word due to the frequency drop phenomenon. Analogous to its application in machine translation, DRDA could potentially mitigate this frequency drop, thus improving the word representation learning.
However, in term of LLMs trained with huge amount of data, the frequency drop phenomenon is observed to be less significant. This is evident from the fact that the larger WMT dataset exhibit a more moderate frequency drop than IWSLT, as shown in Figure 6. Considering this, we believe DRDA is particularly beneficial in a specific downstream scope where there exists rare and complex terminologies, but the training corpus is not sufficient for the model to understood those words. For example, in the context of biology, the term glycosaminoglycan may occur infrrequently. Nevertheless, by leveraging the decomposition of this term into glyco, amino, and glycan, which are more frequently encountered in the corpus, the model can potentially develop a deeper understanding of the term.

heading: Future Work
text: In the future, a better re-ranking approach can be adopted to build a selection model closer to the oracle model