Multilingual Neural Machine Translation With Soft Decoupled Encoding

1 INTRODUCTION . Multilingual Neural Machine Translation ( NMT ) has shown great potential both in creating parameter-efficient MT systems for many languages ( Johnson et al. , 2016 ) , and in improving translation quality of low-resource languages ( Zoph et al. , 2016 ; Firat et al. , 2016 ; Gu et al. , 2018 ; Neubig & Hu , 2018 ; Nguyen & Chiang , 2018 ) . Despite the success of multilingual NMT , it remains a research question how to represent the words from multiple languages in a way that is both parameter efficient and conducive to cross-lingual generalization . The standard sequence-to-sequence ( seq2seq ) NMT model ( Sutskever et al. , 2014 ) represents each lexical unit by a vector from a look-up table , making it difficult to share across different languages with limited lexicon overlap . This problem is particularly salient when translating low-resource languages , where there is not sufficient data to fully train the word embeddings . Several methods have been proposed to alleviate this data sparsity problem in multilingual lexical representation . The current de-facto standard method is to use subword units ( Sennrich et al. , 2016 ; Kudo , 2018 ) , which split up longer words into shorter subwords to allow for generalization across morphological variants or compounds ( e.g . “ un/decide/d ” or “ sub/word ” ) . These can be applied to the concatenated multilingual data , producing a shared vocabulary for different languages , resulting in sharing some but not all subwords of similarly spelled words ( such as “ traducción ” in Spanish and “ tradução ” in Portuguese , which share the root “ tradu- ” ) . However , subword-based preprocessing can produce sub-optimal segmentations for multilingual data , with semantically identical and similarly spelled languages being split into different granularities ( e.g . “ traducción ” and “ tradu/ção ” ) leading to disconnect in the resulting representations . This problem is especially salient when the high-resource language dominates the training data ( see empirical results in Section 4 ) . In this paper , we propose Soft Decoupled Encoding ( SDE ) , a multilingual lexicon representation framework that obviates the need for segmentation by representing words on a full-word level , but can nonetheless share parameters intelligently , aiding generalization . Specifically , SDE softly decouples the traditional word embedding into two interacting components : one component represents 1The source code is available at https : //github.com/cindyxinyiwang/SDE how the word is spelled , and the other component represents the word ’ s latent meaning , which is shared over all languages present at training time . We can view this representation as a decomposition of language-specific realization of the word ’ s form ( i.e . its spelling ) and its language-agnostic semantic function . More importantly , our decoupling is done in a soft manner to preserve the interaction between these two components . SDE has three key components : 1 ) an encoding of a word using character n-grams ( Wieting et al. , 2016 ) ; 2 ) a language specific transform for the character encoding ; 3 ) a latent word embedding constructed by using the character encoding to attend to a shared word embedding space , inspired by Gu et al . ( 2018 ) . Our method can enhance lexical-level transfer through the shared latent word embedding while preserving the model ’ s capacity to learn specific features for each language . Moreover , it eliminates unknown words without any external preprocessing step such as subword segmentation . We test SDE on four low-resource languages from a multilingual TED corpus ( Qi et al. , 2018 ) . Our method shows consistent improvements over multilingual NMT baselines for all four languages , and importantly outperforms previous methods for multilingual NMT that allow for more intelligent parameter sharing but do not use a two-step process of character-level representation and latent meaning representation ( Gu et al. , 2018 ) . Our method outperforms the best baseline by about 2 BLEU for one of the low-resource languages , achieving new state-of-the-art results on all four language pairs compared to strong multi-lingually trained and adapted baselines ( Neubig & Hu , 2018 ) . 2 LEXICAL REPRESENTATION FOR MULTILINGUAL NMT . In this section , we first revisit the 3-step process of computing lexical representations for multilingual NMT , which is illustrated in Figure 1 . Then , we discuss various design choices for each step , as well as the desiderata of an ideal lexical representation for multilingual NMT . 2.1 LEXICAL UNIT SEGMENTATION . The first step to compute the neural representation for a sentence is to segment the sentence into lexical units . There are three popular options with different granularities : • Word-based method splits an input sequence into words , often based on white spaces or punctuation . This is perhaps the natural choice for lexical unit segmentation . Early work in NMT all employ this method ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) . • Character-based method splits an input sequence into characters ( Lee et al. , 2017 ) . • Subword-based method splits each word into pieces from a small vocabulary of frequently occurring patterns ( Sennrich et al. , 2016 ; Kudo , 2018 ) . 2.2 EMBEDDING LOOK-UP . After a sentence is segmented into lexical units , NMT models generally look up embeddings from a dictionary to turn each lexical unit into a high-dimensional vector . In the context of multilingual NMT , the lexical unit segmentation method affects this dictionary in different ways . In word-based segmentation , since the number of unique words for each language is unbounded , while computer memory is not , previous work , e.g . Sutskever et al . ( 2014 ) , resorts to a fixed-size vocabulary of the most frequent words , while mapping out-of-vocabulary words to an 〈unk〉 token . For multilingual NMT settings , where multiple languages are processed , the number of words mapped to 〈unk〉 significantly increases . Moreover , different languages , even related languages , have very few words that have exactly the same spelling , which leads to the same concept being represented by multiple and independent parameters . This disadvantage hurts the translation model ’ s ability to learn the same concept in multiple languages . Meanwhile , character-based segmentation can effectively reduce the vocabulary size , while maximizing the potential for parameter sharing between languages with identical or similar character sets . However , character segmentation is based on the strong assumption that neural networks can infer meaningful semantic boundaries and compose characters into meaningful words . This puts a large amount of pressure on neural models , requiring larger model sizes and training data . Additionally , training character-based NMT systems is often slow , due to the longer character sequences ( Cherry et al. , 2018 ) . Subword-based segmentation is a middle ground between word and character segmentation . However , in multilingual translation , the subword segmentation can be sub-optimal , as the subwords from high-resource languages , i.e . languages with more training data , might dominate the subword vocabulary , so that the words in low-resource language can be split into extremely small pieces . Therefore , existing methods for lexical unit segmentation lead to difficulties in building an effective embedding look-up strategy for multilingual NMT . 2.3 OPTIONAL ENCODING TRANSFORMATIONS . Most commonly , the embedding vectors looked up from the embedding table are used as the final lexical representation . However , it is also possible to have multiple versions of embeddings for a single lexicon and combine them through operations such as attentional weighted sum ( Gu et al. , 2018 ) . Without loss of generality , we can assume there is always a transformation applied to the embedding vectors , and models that do not use such a transformation can be treated as using the identity transformation . 2.4 DESIDERATA . To efficiently utilize parameters for multilingual NMT , the lexical representation should have two properties . First , for maximal accuracy , the lexical representation should be able to accurately represent words in all of the languages under consideration . Second , for better cross-lingual learning and generalization , such a representation should maximize the sharing of parameters across languages . These two conflicting objectives are difficult to achieve through existing methods . The most common method of using lookup embeddings can only share information through lexical units that overlap between the languages . Subword segmentation strikes a middle ground , but has many potential problems for multilingual NMT , as already discussed in Section 2.2 . Although Gu et al . ( 2018 ) ’ s method of latent encoding increases lexical level parameter sharing , it still relies on subwords as its fundamental units , and thus inherits the previously stated problems of sub-word segmentation . We also find in experiments in Section 4 that it is actually less robust than simple lookup when large monolingual data to pre-train embeddings is not available , which is the case for many low-resourced languages . Next , in Section 3 , we propose a novel lexical representation strategy that achieves both desiderata . 3 SOFT DECOUPLED ENCODING . Given the conflict between sharing lexical features and preserving language specific properties , we propose SDE , a general framework to represent lexical units for multilingual NMT . Specifically , following the linguistic concept of the “ arbitrariness of the sign ” ( Chandler , 2007 ) , SDE decomposes the modeling of each word into two stages : ( 1 ) modeling the language-specific spelling of the word , and ( 2 ) modeling the language-agnostic semantics of the word . This decomposition is based on the need for distinguished treatments between a word ’ s semantics and its spelling . Semantic representation is language-agnostic . For example , the English “ hello ” and the French “ bonjour ” deliver the same greeting message , which is invariant with respect to the language . SDE shares such semantic representations among languages by querying a list of shared concepts , which are loosely related to the linguistic concept of “ sememes ” ( Greimas , 1983 ) . This design is implemented using an attention mechanism , where the query is the lexical unit representation , and the keys and the values come from an embedding matrix shared among all languages . Meanwhile , the word spellings are more sophisticated . Here , we identify two important observations about word spellings . First , words in related languages can have similar spellings , e.g . the English word “ color ” and the French word “ coleur ” . In order to effectively share parameters among languages , a word spelling model should utilize this fact . Second , and not contradicting the first point , related languages can also exhibit consistent spelling shifts . For instance , “ Christopher ” , a common name in English , has the spelling “ Kryštof ” in Czech . This necessitates a learnable rule to convert the spelling representations between such pairs of words in related languages . To account for both points , we use a language-specific transformation on top of a first encoding layer based on character n-grams . 3.1 EXISTING METHODS . Before we describe our specific architecture in detail ( Section 3.2 ) , given these desiderata discussed above , we summarize the designs of several existing methods for lexical representation and our proposed SDE framework in Table 1 . Without a preprocessing step of subword segmentation , SDE can capture the lexical similarities of two related languages through the character n-gram embedding while preserving the semantic meaning of lexicons through a shared latent embedding .

Summary: This paper focuses on the problem of word representations in multilingual NMT system. The idea of multilingual NMT is to share data among multiple language pairs. Crucially this requires some way to tie the parameters of words from different languages, and one popular method is to share subword units among languages. The problem is that subword units in different languages may not be semantically equivalent, and many semantically-equivalent concepts are not represented by the same subwords. This paper proposes an alternative way to share word representation, in particular by proposing a common set of "semantic" concept vectors across languages which are then folded into the word representations via attention.