Multilingual Neural Machine Translation With Soft Decoupled Encoding

1 INTRODUCTION . Multilingual Neural Machine Translation ( NMT ) has shown great potential both in creating parameter-efficient MT systems for many languages ( Johnson et al. , 2016 ) , and in improving translation quality of low-resource languages ( Zoph et al. , 2016 ; Firat et al. , 2016 ; Gu et al. , 2018 ; Neubig & Hu , 2018 ; Nguyen & Chiang , 2018 ) . Despite the success of multilingual NMT , it remains a research question how to represent the words from multiple languages in a way that is both parameter efficient and conducive to cross-lingual generalization . The standard sequence-to-sequence ( seq2seq ) NMT model ( Sutskever et al. , 2014 ) represents each lexical unit by a vector from a look-up table , making it difficult to share across different languages with limited lexicon overlap . This problem is particularly salient when translating low-resource languages , where there is not sufficient data to fully train the word embeddings . Several methods have been proposed to alleviate this data sparsity problem in multilingual lexical representation . The current de-facto standard method is to use subword units ( Sennrich et al. , 2016 ; Kudo , 2018 ) , which split up longer words into shorter subwords to allow for generalization across morphological variants or compounds ( e.g . “ un/decide/d ” or “ sub/word ” ) . These can be applied to the concatenated multilingual data , producing a shared vocabulary for different languages , resulting in sharing some but not all subwords of similarly spelled words ( such as “ traducción ” in Spanish and “ tradução ” in Portuguese , which share the root “ tradu- ” ) . However , subword-based preprocessing can produce sub-optimal segmentations for multilingual data , with semantically identical and similarly spelled languages being split into different granularities ( e.g . “ traducción ” and “ tradu/ção ” ) leading to disconnect in the resulting representations . This problem is especially salient when the high-resource language dominates the training data ( see empirical results in Section 4 ) . In this paper , we propose Soft Decoupled Encoding ( SDE ) , a multilingual lexicon representation framework that obviates the need for segmentation by representing words on a full-word level , but can nonetheless share parameters intelligently , aiding generalization . Specifically , SDE softly decouples the traditional word embedding into two interacting components : one component represents 1The source code is available at https : //github.com/cindyxinyiwang/SDE how the word is spelled , and the other component represents the word ’ s latent meaning , which is shared over all languages present at training time . We can view this representation as a decomposition of language-specific realization of the word ’ s form ( i.e . its spelling ) and its language-agnostic semantic function . More importantly , our decoupling is done in a soft manner to preserve the interaction between these two components . SDE has three key components : 1 ) an encoding of a word using character n-grams ( Wieting et al. , 2016 ) ; 2 ) a language specific transform for the character encoding ; 3 ) a latent word embedding constructed by using the character encoding to attend to a shared word embedding space , inspired by Gu et al . ( 2018 ) . Our method can enhance lexical-level transfer through the shared latent word embedding while preserving the model ’ s capacity to learn specific features for each language . Moreover , it eliminates unknown words without any external preprocessing step such as subword segmentation . We test SDE on four low-resource languages from a multilingual TED corpus ( Qi et al. , 2018 ) . Our method shows consistent improvements over multilingual NMT baselines for all four languages , and importantly outperforms previous methods for multilingual NMT that allow for more intelligent parameter sharing but do not use a two-step process of character-level representation and latent meaning representation ( Gu et al. , 2018 ) . Our method outperforms the best baseline by about 2 BLEU for one of the low-resource languages , achieving new state-of-the-art results on all four language pairs compared to strong multi-lingually trained and adapted baselines ( Neubig & Hu , 2018 ) . 2 LEXICAL REPRESENTATION FOR MULTILINGUAL NMT . In this section , we first revisit the 3-step process of computing lexical representations for multilingual NMT , which is illustrated in Figure 1 . Then , we discuss various design choices for each step , as well as the desiderata of an ideal lexical representation for multilingual NMT . 2.1 LEXICAL UNIT SEGMENTATION . The first step to compute the neural representation for a sentence is to segment the sentence into lexical units . There are three popular options with different granularities : • Word-based method splits an input sequence into words , often based on white spaces or punctuation . This is perhaps the natural choice for lexical unit segmentation . Early work in NMT all employ this method ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) . • Character-based method splits an input sequence into characters ( Lee et al. , 2017 ) . • Subword-based method splits each word into pieces from a small vocabulary of frequently occurring patterns ( Sennrich et al. , 2016 ; Kudo , 2018 ) . 2.2 EMBEDDING LOOK-UP . After a sentence is segmented into lexical units , NMT models generally look up embeddings from a dictionary to turn each lexical unit into a high-dimensional vector . In the context of multilingual NMT , the lexical unit segmentation method affects this dictionary in different ways . In word-based segmentation , since the number of unique words for each language is unbounded , while computer memory is not , previous work , e.g . Sutskever et al . ( 2014 ) , resorts to a fixed-size vocabulary of the most frequent words , while mapping out-of-vocabulary words to an 〈unk〉 token . For multilingual NMT settings , where multiple languages are processed , the number of words mapped to 〈unk〉 significantly increases . Moreover , different languages , even related languages , have very few words that have exactly the same spelling , which leads to the same concept being represented by multiple and independent parameters . This disadvantage hurts the translation model ’ s ability to learn the same concept in multiple languages . Meanwhile , character-based segmentation can effectively reduce the vocabulary size , while maximizing the potential for parameter sharing between languages with identical or similar character sets . However , character segmentation is based on the strong assumption that neural networks can infer meaningful semantic boundaries and compose characters into meaningful words . This puts a large amount of pressure on neural models , requiring larger model sizes and training data . Additionally , training character-based NMT systems is often slow , due to the longer character sequences ( Cherry et al. , 2018 ) . Subword-based segmentation is a middle ground between word and character segmentation . However , in multilingual translation , the subword segmentation can be sub-optimal , as the subwords from high-resource languages , i.e . languages with more training data , might dominate the subword vocabulary , so that the words in low-resource language can be split into extremely small pieces . Therefore , existing methods for lexical unit segmentation lead to difficulties in building an effective embedding look-up strategy for multilingual NMT . 2.3 OPTIONAL ENCODING TRANSFORMATIONS . Most commonly , the embedding vectors looked up from the embedding table are used as the final lexical representation . However , it is also possible to have multiple versions of embeddings for a single lexicon and combine them through operations such as attentional weighted sum ( Gu et al. , 2018 ) . Without loss of generality , we can assume there is always a transformation applied to the embedding vectors , and models that do not use such a transformation can be treated as using the identity transformation . 2.4 DESIDERATA . To efficiently utilize parameters for multilingual NMT , the lexical representation should have two properties . First , for maximal accuracy , the lexical representation should be able to accurately represent words in all of the languages under consideration . Second , for better cross-lingual learning and generalization , such a representation should maximize the sharing of parameters across languages . These two conflicting objectives are difficult to achieve through existing methods . The most common method of using lookup embeddings can only share information through lexical units that overlap between the languages . Subword segmentation strikes a middle ground , but has many potential problems for multilingual NMT , as already discussed in Section 2.2 . Although Gu et al . ( 2018 ) ’ s method of latent encoding increases lexical level parameter sharing , it still relies on subwords as its fundamental units , and thus inherits the previously stated problems of sub-word segmentation . We also find in experiments in Section 4 that it is actually less robust than simple lookup when large monolingual data to pre-train embeddings is not available , which is the case for many low-resourced languages . Next , in Section 3 , we propose a novel lexical representation strategy that achieves both desiderata . 3 SOFT DECOUPLED ENCODING . Given the conflict between sharing lexical features and preserving language specific properties , we propose SDE , a general framework to represent lexical units for multilingual NMT . Specifically , following the linguistic concept of the “ arbitrariness of the sign ” ( Chandler , 2007 ) , SDE decomposes the modeling of each word into two stages : ( 1 ) modeling the language-specific spelling of the word , and ( 2 ) modeling the language-agnostic semantics of the word . This decomposition is based on the need for distinguished treatments between a word ’ s semantics and its spelling . Semantic representation is language-agnostic . For example , the English “ hello ” and the French “ bonjour ” deliver the same greeting message , which is invariant with respect to the language . SDE shares such semantic representations among languages by querying a list of shared concepts , which are loosely related to the linguistic concept of “ sememes ” ( Greimas , 1983 ) . This design is implemented using an attention mechanism , where the query is the lexical unit representation , and the keys and the values come from an embedding matrix shared among all languages . Meanwhile , the word spellings are more sophisticated . Here , we identify two important observations about word spellings . First , words in related languages can have similar spellings , e.g . the English word “ color ” and the French word “ coleur ” . In order to effectively share parameters among languages , a word spelling model should utilize this fact . Second , and not contradicting the first point , related languages can also exhibit consistent spelling shifts . For instance , “ Christopher ” , a common name in English , has the spelling “ Kryštof ” in Czech . This necessitates a learnable rule to convert the spelling representations between such pairs of words in related languages . To account for both points , we use a language-specific transformation on top of a first encoding layer based on character n-grams . 3.1 EXISTING METHODS . Before we describe our specific architecture in detail ( Section 3.2 ) , given these desiderata discussed above , we summarize the designs of several existing methods for lexical representation and our proposed SDE framework in Table 1 . Without a preprocessing step of subword segmentation , SDE can capture the lexical similarities of two related languages through the character n-gram embedding while preserving the semantic meaning of lexicons through a shared latent embedding .

Summary: This paper presents an approach to creating word representations that operate at both the sub-word level and generalise across languages. The paper presents soft decoupled encoding as a method to learn word representations from weighted bags of character-n grams, a language specific transformation layer, and a "latent semantic embedding" layer. The experiments are conducted over low-resource languages from the multilingual TED corpus. The experiments show consistent improvements compared to existing approaches to training translation models with sub-word representations. The ablation studies in Section 4.4 are informative about the relative importance of different parts of the proposed model.