Introduction
The term, related languages, refers to languages that exhibit lexical and structural similarities on account of sharing a common ancestry or being in contact for a long period of time BIBREF0 . Examples of languages related by common ancestry are Slavic and Indo-Aryan languages. Prolonged contact leads to convergence of linguistic properties even if the languages are not related by ancestry and could lead to the formation of linguistic areas BIBREF1 . Examples of such linguistic areas are the Indian subcontinent BIBREF2 , Balkan BIBREF3 and Standard Average European BIBREF4 linguistic areas. Genetic as well as contact relationship lead to related languages sharing vocabulary and structural features.
There is substantial government, commercial and cultural communication among people speaking related languages (Europe, India and South-East Asia being prominent examples and linguistic regions in Africa possibly in the future). As these regions integrate more closely and move to a digital society, translation between related languages is becoming an important requirement. In addition, translation to/from related languages to a lingua franca like English is also very important. However, despite significant communication between people speaking related languages, most of these languages have few parallel corpora resources. It is therefore important to leverage the relatedness of these languages to build good-quality statistical machine translation (SMT) systems given the lack of parallel corpora.
Modelling lexical similarity among related languages is the key to building good-quality SMT systems with limited parallel corpora. Lexical similarity implies related languages share many words with similar form (spelling/pronunciation) and meaning e.g. blindness is andhapana in Hindi, aandhaLepaNaa in Marathi. These words could be cognates, lateral borrowings or loan words from other languages.
Subword level transformations are an effective way for translation of such shared words. In this work, we propose use of Byte Pair Encoding (BPE) BIBREF5 , BIBREF6 , a encoding method inspired from text compression literature, to learn basic translation units for translation between related languages. In previous work, the basic units of translation are either linguistically motivated (word, morpheme, syllable, etc.) or ad-hoc choices (character n-gram). In contrast, BPE is motivated by statistical properties of text.
The major contributions of our work are:
The paper is organized as follows. Section SECREF2 discusses related work. Section SECREF3 discusses why BPE is a promising method for learning subword units and describes how we train BPE unit level translation models. Section SECREF4 describes our experimental set-up. Section SECREF5 reports the results of our experiments and analyses the results. Based on experimental results, we analyse why BPE units outperform other units in Section SECREF6 . Section SECREF7 concludes the paper by summarizing our work and discussing further research directions.
Related Work
There are two broad set of approaches that have been explored in the literature for translation between related languages that leverage lexical similarity between source and target languages.
The first approach involves transliteration of source words into the target languages. This can done by transliterating the untranslated words in a post-processing step BIBREF8 , BIBREF9 , a technique generally used for handling named entities in SMT. However, transliteration candidates cannot be scored and tuned along with other features used in the SMT system. This limitation can be overcome by integrating the transliteration module into the decoder BIBREF10 , so both translation and transliteration candidates can be evaluated and scored simultaneously. This also allows transliteration vs. translation choices to be made.
Since a high degree of similarity exists at the subword level between related languages, the second approach looks at translation with subword level basic units. Character-level SMT has been explored for very closely related languages like Bulgarian-Macedonian, Indonesian-Malay, Spanish-Catalan with modest success BIBREF11 , BIBREF12 , BIBREF13 . Unigram-level learning provides very little context for learning translation models BIBREF14 . The use of character n-gram units to address this limitation leads to data sparsity for higher order n-grams and provides little benefit BIBREF13 . These results were demonstrated primarily for very close European languages. kunchukuttan2016orthographic proposed orthographic syllables, a linguistically-motivated variable-length unit, which approximates a syllable. This unit has outperformed character n-gram, word and morpheme level models as well as transliteration post-editing approaches mentioned earlier. They also showed orthographic syllables can outperform other units even when: (i) the lexical distance between related languages is reasonably large, (ii) the languages do not have a genetic relation, but only a contact relation.
Recently, subword level models have also generated interest for neural machine translation (NMT) systems. The motivation is the need to limit the vocabulary of neural MT systems in encoder-decoder architectures BIBREF15 . It is in this context that Byte Pair Encoding, a data compression method BIBREF5 , was adapted to learn subword units for NMT BIBREF6 . Other subword units for NMT have also been proposed: character BIBREF16 , Huffman encoding based units BIBREF17 , wordpieces BIBREF18 , BIBREF19 . Our hypothesis is that such subword units learnt from corpora are particularly suited for translation between related languages. In this paper, we test this hypothesis by using BPE to learn subword units.
BPE for related languages
We discuss why BPE is a promising method for learning subword units (subsections SECREF7 and SECREF8 ) and describe how we trained our BPE unit level translation models (subsections SECREF9 and SECREF10 ).
Motivation
Byte Pair Encoding is a data compression algorithm which was first adapted for Neural Machine Translation by sennrich2016neural. For a given language, it is used to build a vocabulary relevant to translation by discovering the most frequent character sequences in the language.
For NMT, BPE enables efficient, high quality, open vocabulary translation by (i) limiting core vocabulary size, (ii) representing the most frequent words as atomic BPE units and rare words as compositions of the atomic BPE units. These benefits of BPE are not particular to NMT, and apply to SMT between related languages too. Given the lexical similarity between related languages, we would like to identify a small, core vocabulary of subwords from which words in the language can be composed. These subwords represent stable, frequent patterns (possibly linguistic units like syllables, morphemes, affixes) for which mappings exist in other related languages. This alleviates the need for word level translation.
Comparison with orthographic syllables
We primarily compare BPE units with orthographic syllables (OS) BIBREF7 , which are good translation units for related languages. The orthographic syllable is a sequence of one or more consonants followed by a vowel, i.e. a C INLINEFORM0 V unit, which approximates a linguistic syllable (e.g. spacious would be segmented as spa ciou s). Orthographic syllabification is rule based and applies to writing systems which represent vowels (alphabets and abugidas).
Both OS and BPE units are variable length units which provide longer and more relevant context for translation compared to character n-grams. In contrast to orthographic syllables, the BPE units are highly frequent character sequences reflecting the underlying statistical properties of the text. Some of the character sequences discovered by the BPE algorithm may be different linguistic units like syllables, morphemes and affixes. Moreover, BPE can be applied to text in any writing system.
The BPE Algorithm
We briefly summarize the BPE algorithm (described at length in sennrich2016neural). The input is a monolingual corpus for a language (one side of the parallel training data, in our case). We start with an initial vocabulary viz. the characters in the text corpus. The vocabulary is updated using an iterative greedy algorithm. In every iteration, the most frequent bigram (based on current vocabulary) in the corpus is added to the vocabulary (the merge operation). The corpus is again encoded using the updated vocabulary and this process is repeated for a pre-determined number of merge operations. The number of merge operations is the only hyperparameter to the system which needs to be tuned. A new word can be segmented by looking up the learnt vocabulary. For instance, a new word scion may be segmented as sc ion after looking up the learnt vocabulary, assuming sc and ion as BPE units learnt during training.
Training subword level translation model
We train subword level phrase-based SMT models between related languages. Along with BPE level, we also train PBSMT models at morpheme and OS levels for comparison.
For BPE, we learn the vocabulary separately for the source and target languages using the respective part of the training corpus. We segment the data into subwords during pre-processing and indicate word boundaries by a boundary marker (_) as shown in the example below. The boundary marker helps keep track of word boundaries, so the word level representation can be reconstructed after decoding.
word: Childhood means simplicity .
subword: Chi ldhoo d _ mea ns _ si mpli ci ty _ .
While building phrase-based SMT models at the subword level, we use (a) monotonic decoding since related languages have similar word order, (b) higher order languages models (10-gram) since data sparsity is a lesser concern owing to small vocabulary size BIBREF11 , and (c) word level tuning (by post-processing the decoder output during tuning) to optimize the correct translation metric BIBREF8 . Following decoding, we used a simple method to regenerate words from subwords (desegmentation): concatenate subwords between consecutive occurrences of boundary marker characters.
Experimental Setup
We trained translation systems over the following basic units: character, morpheme, word, orthographic syllable and BPE unit. In this section, we summarize the languages and writing systems chosen for our experiments, the datasets used and the experimental configuration of our translation systems, and the evaluation methodology.
Languages and writing systems
Our experiments spanned a diverse set of languages: 16 language pairs, 17 languages and 10 writing systems. Table TABREF11 summarizes the key aspects of the languages involved in the experiments.
The chosen languages span 4 major language families (6 major sub-groups: Indo-Aryan, Slavic and Germanic belong to the larger Indo-European language family). The languages exhibit diversity in word order and morphological complexity. Of course, between related languages, word order and morphological properties are similar. The classification of Japanese and Korean into the Altaic family is debated, but various lexical and grammatical similarities are indisputable, either due to genetic or cognate relationship BIBREF20 , BIBREF21 . However, the source of lexical similarity is immaterial to the current work. For want of a better classification, we use the name Altaic to indicate relatedness between Japanese and Korean.
The chosen language pairs also exhibit varying levels of lexical similarity. Table TABREF22 shows an indication of the lexical similarity between them in terms of the Longest Common Subsequence Ratio (LCSR) BIBREF22 . The LCSR has been computed over the parallel training sentences at character level (shown only for language pairs where the writing systems are the same or can be easily mapped in order to do the LCSR computation). At one end of the spectrum, Malayalam-India, Urdu-Hindi, Macedonian-Bulgarian are dialects/registers of the same language and exhibit high lexical similarity. At the other end, pairs like Hindi-Malayalam belong to different language families, but show many lexical and grammatical similarities due to contact for a long time BIBREF23 .
The chosen languages cover 5 types of writing systems. Of these, alphabetic and abugida writing systems represent vowels, logographic writing systems do not have vowels. The use of vowels is optional in abjad writing systems and depends on various factors and conventions. For instance, Urdu word segmentation can be very inconsistent BIBREF24 and generally short vowels are not denoted. The Korean Hangul writing system is syllabic, so the vowels are implicitly represented in the characters.
Datasets
Table TABREF14 shows train, test and tune splits of the parallel corpora used. The Indo-Aryan and Dravidian language parallel corpora are obtained from the multilingual Indian Language Corpora Initiative (ILCI) corpus BIBREF25 . Parallel corpora for other pairs were obtained from the OpenSubtitles2016 section of the OPUS corpus collection BIBREF26 . Language models for word-level systems were trained on the target side of training corpora plus additional monolingual corpora from various sources (See Table TABREF14 for details). We used just the target language side of the parallel corpora for character, morpheme, OS and BPE-unit level LMs.
System details
We trained phrase-based SMT systems using the Moses system BIBREF31 , with the grow-diag-final-and heuristic for extracting phrases, and Batch MIRA BIBREF32 for tuning (default parameters). We trained 5-gram LMs with Kneser-Ney smoothing for word and morpheme level models and 10-gram LMs for character, OS and BPE-unit level models. Subword level representation of sentences is long, hence we speed up decoding by using cube pruning with a smaller beam size (pop-limit=1000). This setting has been shown to have minimal impact on translation quality BIBREF33 .
We used unsupervised morphological-segmenters for generating morpheme representations (trained using Morfessor BIBREF34 ). For Indian languages, we used the models distributed as part of the Indic NLP Library BIBREF9 . We used orthographic syllabification rules from the Indic NLP Library for Indian languages, and custom rules for Latin and Slavic scripts. For training BPE models, we used the subword-nmt library. We used Juman and Mecab for Japanese and Korean tokenization respectively.
For mapping characters across Indic scripts, we used the method described by kunchukuttan2015brahmi and implemented in the Indic NLP Library.
Evaluation
The primary evaluation metric is word-level BLEU BIBREF35 . We also report LeBLEU BIBREF36 scores as an alternative evaluation metric. LeBLEU is a variant of BLEU that does an edit-distance based, soft-matching of words and has been shown to be better for morphologically rich languages. We used bootstrap resampling for testing statistical significance BIBREF37 .
Results and Analysis
This section describes the results of various experiments and analyses them. A comparison of BPE with other units across languages and writing systems, choice of number of merge operations and effect of domain change and training data size are studied. We also report initial results with a joint bilingual BPE model.
Comparison of BPE with other units
Table TABREF22 shows translation accuracies of all the language pairs under experimentation for different translation units, in terms of BLEU as well as LeBLEU scores. The number of BPE merge operations was chosen such that the resultant vocabulary size would be equivalent to the vocabulary size of the orthographic syllable encoded corpus. Since we could not do orthographic syllabification for Urdu, Korean and Japanese, we selected the merge operations as follows: For Urdu, number of merge operations were selected based on Hindi OS vocabulary since Hindi and Urdu are registers of the same language. For Korean and Japanese, the number of BPE merge operations was set to 3000, discovered by tuning on a separate validation set.
Our major observations are described below (based on BLEU scores):
INLINEFORM0 BPE units are clearly better than the traditional word and morpheme representations. The average BLEU score improvement is 15% over word-based results and 11% over morpheme-based results. The only exception is Malay-Indonesian, which are registers of the same language.
INLINEFORM0 BPE units also show modest improvement over the recently proposed orthographic syllables over most language pairs (average improvement of 2.6% and maximum improvement of up to 11%). The improvements are not statistically significant for most language pairs. The only exceptions are Bengali-Hindi, Punjabi-Hindi and Malay-Indonesian - all these languages pairs have relatively less morphological affixing (Bengali-Hindi, Punjabi-Hindi) or are registers of the same language (Malay-Indonesian). For Bengali-Hindi and Punjabi-Hindi, the BPE unit translation accuracies are quite close to OS level accuracies. Since OS level models have been shown to be better than character level models BIBREF7 , BPE units are better than character level models by transitivity.
INLINEFORM0 BPE units also outperform other units for translation between language pairs belonging to different language pairs, but having a long contact relationship viz. Malayalam-Hindi and Hindi-Malayalam.
INLINEFORM0 It is worth mentioning that BPE units provide a substantial benefit over OS units when translation involves a morphologically rich language. In translations involving Malayalam, Tamil and Telugu, average accuracy improvement of 6.25% were observed.
The LeBLEU scores also show the same trends as the BLEU scores.
Applicability to different writing systems
The utility of orthographic syllables as translation units is limited to languages that use writing systems which represent vowels. Alphabetic and abugida writing systems fall into this category. On the other hand, logographic writing systems (Japanese Kanji, Chinese) and abjad writing systems (Arabic, Hebrew, Syriac, etc.) do not represent vowels. To be more precise, abjad writing systems may represent some/all vowels depending on language, pragmatics and conventions. Syllabic writing systems like Korean Hangul do not explicitly represent vowels, since the basic unit (the syllable) implicitly represents the vowels. The major advantage of Byte Pair Encoding is its writing system independence and our results show that BPE encoded units are useful for translation involving abjad (Urdu uses an extended Arabic writing system), logographic (Japanese Kanji) and syllabic (Korean Hangul) writing systems. For language pairs involving Urdu, there is an 18% average improvement over word-level and 12% average improvement over morpheme-level translation accuracy. For Japanese-Korean language pairs, an average improvement of 6% in translation accuracy over a word-level baseline is observed.
Choosing number of BPE merges
The above mentioned results for BPE units do not explore optimal values of the number of merge operations. This is the only hyper-parameter that has to be selected for BPE. We experimented with number of merge operations ranging from 1000 to 4000 and the translation results for these are shown in Table TABREF25 . Selecting the optimal value of merge operations lead to a modest, average increase of 1.6% and maximum increase of 3.5% in the translation accuracy over B INLINEFORM0 across different language pairs .
We also experimented with higher number of merge operations for some language pairs, but there seemed to be no benefit with a higher number of merge operations. Compared to the number of merge operations reported by sennrich2016neural in a more general setting for NMT (60k), the number of merge operations is far less for translation between related languages with limited parallel corpora. We must bear in mind that their goal was different: available parallel corpus was not an issue, but they wanted to handle as large a vocabulary as possible for open-vocabulary NMT. Yet, the low number of merge operations suggest that BPE encoding captures the core vocabulary required for translation between related tasks.
Robustness to Domain Change
Since we are concerned with low resource scenarios, a desirable property of subword units is robustness of the translation models to change of translation domain. kunchukuttan2016orthographic have shown that OS level models are robust to domain change. Since BPE units are learnt from a specific corpus, it is not guaranteed that they would also be robust to domain changes. To study the behaviour of BPE unit trained models, we also tested the translation models trained on tourism & health domains on an agriculture domain test set of 1000 sentences (see Table TABREF27 for results). In this cross-domain translation scenario, the BPE level model outperforms the OS-level and word-level models for most language pairs. The Konkani-Marathi pair alone shows a degradation using the OS level model. The BPE model is almost on par with the OS level model for Telugu-Malayalam and Hindi-Malayalam.
Effect of training data size
For different training set sizes, we trained SMT systems with various representation units (Figure FIGREF28 shows the learning curves for two language pairs). BPE level models are better than OS, morpheme and word level across a range of dataset sizes. Especially when the training data is very small, the OS and BPE level models perform significantly better than the word and morpheme level models. For Malayalam-Hindi, the BPE level model is better than the OS level model at utilizing more training data.
Joint bilingual learning of BPE units
In the experiments discussed so far, we learnt the BPE vocabulary separately for the source and target languages. In this section, we describe our experiments with jointly learning BPE vocabulary over source and target language corpora as suggested by sennrich2016neural. The idea is to learn an encoding that is consistent across source and target languages and therefore helps alignment. We expect a significant number of common BPE units between related languages. If source and target languages use the same writing system, then a joint model is created by learning BPE over concatenated source and target language corpus. If the writing systems are different, then we transliterate one corpus to another by one-one character mappings. This is possible between Indic scripts. But this scheme cannot be applied between Urdu and Indic scripts as well as between Korean Hangul and Japanese Kanji scripts.
Table TABREF31 shows the results of the joint BPE model for language pairs where such a model is built. We do not see any major improvement over the monolingual BPE model due to the joint BPE model.
Why are BPE units better than others?
The improved performance of BPE units compared to word-level and morpheme-level representations is easy to explain: with a limited vocabulary they address the problem of data sparsity. But character level models also have a limited vocabulary, yet they do not improve translation performance except for very close languages. Character level models learn character mappings effectively, which is sufficient for translating related languages which are very close to each other (translation is akin to transliteration in these cases). But they are not sufficient for translating related languages that are more divergent. In this case, translating cognates, morphological affixes, non-cognates etc. require a larger context. So, BPE and OS units — which provide more context — outperform character units.
A study of the correlation between lexical similarity and translation quality makes this evident (See Table TABREF32 ). We see that character models work best when the source and target sentences are lexically very similar. The additional context decouples OS and BPE units from lexical similarity. Words and morphemes show the least correlation since they do not depend on lexical similarity.
Why does BPE performs better than OS which also provides a larger contextual window for translation? While orthographic syllables represent just approximate syllables, we observe that BPE units also represent higher level semantic units like frequent morphemes, suffixes and entire words. Table TABREF33 shows a few examples for some Indian languages. So, BPE level models can learn semantically similar translation mappings in addition to lexically similar mappings. In this way, BPE units enable the translation models to balance the use of lexical similarity with semantic similarity. This further decouples the translation quality from lexical similarity as seen from the table. BPE units also have an additional degree of freedom (choice of vocabulary size), which allows tuning for best translation performance. This could be important when larger parallel corpora are available, allowing larger vocabulary sizes.
Conclusion & Future Work
We show that translation units learnt using BPE can outperform all previously proposed translation units, including the best-performing orthographic syllables, for SMT between related languages when limited parallel corpus is available. Moreover, BPE encoding is writing system independent, hence it can be applied to any language. Experimentation on a large number of language pairs spanning diverse language families and writing systems lend strong support to our results. We also show that BPE units are more robust to change in translation domain. They perform better for morphologically rich languages and extremely data scarce scenarios.
BPE seems to be beneficial because it enables discovery of translation mappings at various levels simultaneously (syllables, suffixes, morphemes, words, etc. ). We would like to further pursue this line of work and investigate better translation units. This is also a question relevant to translation with subwords in NMT. NMT between related languages using BPE and similar encodings is also an obvious direction to explore.
Given the improved performance of the BPE-unit, tasks involving related languages viz. pivot based MT, domain adaptation BIBREF14 and translation between a lingua franca and related languages BIBREF38 can be revisited with BPE units.
Acknowledgments
We thank the Technology Development for Indian Languages (TDIL) Programme and the Department of Electronics & Information Technology, Govt. of India for their support. We also thank the reviewers for their feedback.

Question: Which translation model do they employ?
Answer: BPE level, PBSMT models at morpheme and OS levels

subword level phrase-based SMT model

Question: Which datasets do they experiment on?
Answer: Indian Language Corpora Initiative (ILCI) corpus, OpenSubtitles2016 section of the OPUS corpus

multilingual Indian Language Corpora Initiative (ILCI) corpus BIBREF25, OpenSubtitles2016 section of the OPUS corpus collection BIBREF26

Question: Which other units of text do they experiment with (apart from BPE and ortographic syllables)?
Answer: character,  morpheme, word

character, morpheme, word

Question: How many steps of BPE do they experiment with?
Answer: from 1000 to 4000

Unanswerable