A Latent Morphology Model for Open-Vocabulary Neural Machine Translation

1 INTRODUCTION . Neural machine translation ( NMT ) models are conventionally trained by maximizing the likelihood of generating the target side of a bilingual parallel corpus of observations one word at a time conditioned of their full observed context . NMT models must therefore learn distributed representations that accurately predict word forms in very diverse contexts , a process that is highly demanding in terms of training data as well as the network capacity . Under conditions of lexical sparsity , which includes both the case of unknown words and the case of known words occurring in surprising contexts , the model is likely to struggle . Such adverse conditions are typical of translation involving morphologically-rich languages , where any single root may lead to exponentially many different surface realizations depending on its syntactic context . Such highly productive processes of word formation lead to many word forms being rarely or ever observed with a particular set of morphosyntactic attributes . The standard approach to overcome this limitation is to pre-process words into subword units that are shared among words , which are , in principle , more reliable as they are observed more frequently in varying context ( Sennrich et al. , 2016 ; Wu et al. , 2016 ) . One drawback related to this approach , however , is that the estimation of the subword vocabulary relies on word segmentation methods optimized using corpus-dependent statistics , disregarding any linguistic notion of morphology and the translation objective . This often produces subword units that are semantically ambiguous as they might be used in far too many lexical and syntactic contexts ( Ataman et al. , 2017 ) . Moreover , in this approach , a word form is then generated by prediction of multiple subword units , which makes generalizing to unseen word forms more difficult due to the possibility that a subword unit necessary to reconstruct a given word form may be unlikely in a given context . To alleviate the sub-optimal effects of using explicit segmentation and generalize better to new morphological forms , recent studies explored the idea of extending NMT to model translation directly at ∗Work done while the first author was a doctoral student at the University of Trento and a visiting postgraduate student at the University of Edinburgh . the level of characters ( Kreutzer & Sokolov , 2018 ; Cherry et al. , 2018 ) , which , in turn , have demonstrated the requirement of using comparably deeper networks , as the network would then need to learn longer distance grammatical dependencies ( Sennrich , 2017 ) . In this paper , we explore the benefits of explicitly modeling variation in surface forms of words using techniques from deep latent variable modeling in order to improve translation accuracy for low-resource and morphologically-rich languages . Latent variable models allow us to inject inductive biases relevant to the task , which , in our case , is word formation during translation . In order to formulate the process of morphological inflection , design a hierarchical latent variable model which translates words one character at a time based on word representations learned compositionally from sub-lexical components . In particular , for each word , our model generates two latent representations : i ) a continuous-space dense vector aimed at capturing the lexical semantics of the word in a given context , and ii ) a set of ( approximately ) discrete features aimed at capturing that word ’ s morphosyntatic role in the sentence . We then see inflection as decoding a word form , one character at a time , from a learned composition of these two representations . By forcing the model to encode each word representation in terms of a more compact set of latent features , we encourage them to be shared across contexts and word forms , thus , facilitating generalization under sparse settings . We evaluate our method in translating English into three morphologically-rich languages each with a distinct morphological typology : Arabic , Czech and Turkish , and show that our model is able to obtain better translation accuracy and generalization capacity than conventional approaches to open-vocabulary NMT . 2 NEURAL MACHINE TRANSLATION . In this paper , we use recurrent NMT architectures based on the model developed by Bahdanau et al . ( 2014 ) . The model essentially estimates the conditional probability of translating a source sequence x = 〈x1 , x2 , . . . xm〉 into a target sequence y = 〈y1 , y2 , . . . yl〉 via an exact factorization : p ( y|x , θ ) = l∏ i=1 p ( yj |x , y < i , θ ) ( 1 ) where y < i stands for the sequence preceding the ith target word . At each step of the sequence , a fixed neural network architecture maps its inputs , the source sentence and the target prefix , to the probability of the ith target word observation in context . In order to condition on the source sentence fully , this network employs an embedding layer and a bi-directional recurrent neural network ( bi-RNN ) based encoder . Conditioning on the target prefix y < i is implemented using a recurrent neural network ( RNN ) based decoder , and an attention mechanism which summarises the source sentence into a context vector ci as a function of a given prefix ( Luong et al. , 2015 ) . Given a parallel training set D , the parameters θ of the network are estimated to attain a local minimum of the negative log-likelihood function L ( θ|D ) = − ∑ x , y∼D log p ( y|x , θ ) via stochastic gradient-based optimization ( Bottou & Cun , 2004 ) . Atomic parameterization estimates the probability of generating each target word yi in a single shot : p ( yi|x , y < i , θ ) = exp ( E > yihi ) ∑v e=1 exp ( E > e hi ) , ( 2 ) where E ∈ Rv×d is the target embedding matrix and the decoder output hi ∈ Rd represents x and y < i . Clearly , the size v of the target vocabulary plays an important role in determining the complexity of the model , which creates an important bottleneck when translating into low-resource and morphologically-rich languages due to the sparsity in the lexical distribution . Recent studies approached this problem by performing NMT with subword units , a popular one of which is based on the Byte-Pair Encoding algorithm ( BPE ; Sennrich et al. , 2016 ) , which finds the optimal description of a corpus vocabulary by iteratively merging the most frequent character sequences . Atomic parameterization could also be used to model translation at the level of characters , which is found to be advantageous in generalizing to morphological variations ( Cherry et al. , 2018 ) . Hierarchical paramaterization further factorizes the probability of a target word in context : p ( yi|x , y < i , θ ) = li∏ j=1 p ( yi , j |x , y < i , yi , < j , θ ) ( 3 ) where the ith word yi = 〈yi,1 , . . . , yi , li〉 is seen as a sequence of li characters . Generation follows one character at a time , each with probability computed by a fixed neural network architecture with varying inputs , namely , the source sentence x , the target prefix y < i , and the prefix yi , < j of characters already generated for that word . In this case there are two recurrent cells , one updated at the boundary of each token , much like in the standard case , and another updated at the character level . Luong & Manning ( 2016 ) propose hierarchical parameterization to compute the probability p ( yi|x , y < i , θ ) for unknown words , while for known words they use the atomic parameterization . In this paper , we use the hierarchical parameterization method for generating all target words , where we also augment the input embedding layer with a character-level bi-RNN , which computes each word representation yi as a composition of the embeddings of their characters ( Ling et al. , 2015 ) . 3 A LATENT MORPHOLOGY MODEL ( LMM ) FOR LEARNING WORD REPRESENTATIONS . The application of a hierarchical structure for learning word representations in language modeling ( Vania & Lopez , 2017 ) or semantic role labeling ( Sahin & Steedman , 2018 ) have shown that such representations encode many cues about the morphological features of words by establishing a mapping between phonetic units and lexical context . Although it can provide an alternative solution to open-vocabulary NMT by potentially alleviating the need for subword segmentation , the quality of word representations learned by an hierarchical model is still highly dependant on the amount of observations ( Sahin & Steedman , 2018 ; Ataman et al. , 2019 ) , since the training data is essential in properly modeling the lexical distribution . On the other hand , the process of word formation , particularly morphological inflection , has many properties that remain universal across languages , where a word is typically composed of a lemma , representing its lexical semantics , and a distinct combination of categorical inflectional features expressing the word ’ s syntactic role in the phrase or sentence . In this paper , we propose to manipulate this universal structure in order to enforce an inductive bias on the prior distribution of words and allow the hierarchical parameterization model in properly learning lexical representations under conditions of data sparsity . 3.1 GENERATIVE MODEL . Our generative LMM for NMT formulates word formation in terms of a stochastic process , where each word is generated one character at a time by composing two latent representations : a continuous vector aimed at capturing the lemma , and a set of sparse features aimed at capturing the inflectional features . The motivation for using a stochastic model is twofold . First , deterministic models are by definition unimodal : when presented with the same input ( the same context ) they always produce the same output . When we model the word formation process , it is reasonable to expect a larger degree of ambiguity , that is , for the same context ( e.g . a noun prefix ) , we may continue by inflecting the word differently depending on the ( latent ) mode of operation we are at ( e.g . generating nominative , accusative or dative noun ) . Second , in stochastic models , the choice of distribution gives us a mechanism to favour a particular type of representation . In our case , we use sparse distributions for inflectional features to accommodate the fact that morphosyntactic features are discrete in nature . Our latent variable model is an instance of a variational auto-encoder ( VAE ; Kingma & Welling , 2013 ) inspired by the model of Zhou & Neubig ( 2017 ) for morphological reinflection . Generation of the ith word starts by sampling a Gaussian-distributed representation in context . This requires predicting the Gaussian location ui and scale si vectors,1 Zi|x , y < i ∼ N ( ui , diag ( si si ) ) ui = dense ( hi ; θu ) si = ζ ( dense ( hi ; θs ) ) ( 4 ) 1Notation We use capital Roman letters for random variables ( and lowercase letters for assignments ) . Boldface Roman letters are reserved for neural network output vectors , and stands for elementwise multiplication . Finally , we denote typical neural network layers as layer ( inputs ; parameters ) . where prediction of the location ( in Rd ) and scale ( in Rd > 0 ) from the word-level decoder hidden state hi ( which represents x and y < i ) is performed by two dense layers , and the scale values are ensured to be positive with the softplus ( ζ ) activation.2 Generation proceeds by then sampling aK-dimensional vector fi of sparse scalar features ( see §3.2 ) conditioned on the source x , the target prefix y < i , and the sampled lemma zi . We model sampling of fi conditioned on zi in order to capture the insight that inflectional transformations typically depend on the category of a lemma . Having sampled fi and zi , the representation of the ith target word is computed by a transformation of zi and fi , i.e . ti = dense ( [ zi , fi ] ; θcomp ) . As shown in Figure 1 , our model generates each word character by character auto-regressively by conditioning on the word representation ti predicted by the LMM , the current context ci , and the previously generated characters following the hierarchical parameterization.3 See Algorithm 1 for details on generation . Input : model parameters θ , latent lemma zi , latent morphological attributes fi , observed character sequence 〈yi,1 , . . . , yi , li〉 if training or placeholders if test , decoder state hi , and context vector ci Result : updated decoder hidden state , prediction ( a word ) , probability of prediction ( for loss ) initialization ; ti = dense ( [ zi , fi ] ; θcomp ) ; initialize char-rnn with a projection of [ ti , ci ] ; for j < li and j < max do compute output layer from char-rnn state ; if training then set prediction to observation yi , j ; else set prediction to argmax of output softmax layer ; end assess log-probability of prediction ; update word-level RNN decoder with prediction ; end Algorithm 1 : Word generation : in training the word is observed , thus we only update the decoder and assess the probability of the observation , in test , we use mean values of the distributions to represent most likely values for z and f and populate predictions with beam-search . 2In practice , we sample zi via a reparameterization in terms of a fixed Gaussian , namely , zi = ui + i si for i ∼ N ( 0 , Id ) . This is known as the reparameterization trick ( Kingma & Welling , 2013 ) , which allows back-propagation through stochastic units ( Rezende et al. , 2014 ) . 3Formally , because the decoder is an RNN , we are also conditioning on z < i and f < i . We omit this dependence to avoid clutter .

Summary: This paper proposes a method, Latent Morphology Model (LMM), for producing word representations for a (hierarchical) character-level decoder used in neural machine translation (NMT). The main motivation is to overcome vocabulary sparsity or highly inflectional languages such as Arabic, Czech, and Turkish (experimented in the paper). To model the morphological inflection, they decouple lemmas and inflection types into 2 latent variables (z and f) where f is enforced to be sparse (arguably mimic the process of the human). The literature review of NMT and the discussion on the potential advantage of morphology are concise. The proposed model is a variation of Luong & Manning (2016) and Schulz et al (2018) models, thus, their main contribution is an introduction of the latent morphological features to the decoder. The proposed LMM is trained by sampling z and f from prior directly, and the sparsity of the morphological features is encouraged by L0 of the feature vector (parameterized as independent Kumaraswamy variables). They perform the main empirical study using 3 languages by translating from English to justify the proposed LMM. Lastly, they provide a quantitative analysis of the perplexities of unseen words and a qualitative on words generated of a lemma with different feature vectors.