A LATENT MORPHOLOGY MODEL FOR OPEN- VOCABULARY NEURAL MACHINE TRANSLATION
Year: 2020

Abstract:
Translation into morphologically-rich languages challenges neural machine translation (NMT) models with extremely sparse vocabularies where atomic treatment of surface forms is unrealistic. This problem is typically addressed by either pre-processing words into subword units or performing translation directly at the level of characters. The former is based on word segmentation algorithms optimized using corpus-level statistics with no regard to the translation task. The latter learns directly from translation data but requires rather deep architectures. In this paper, we propose to translate words by modeling word formation through a hierarchical latent variable model which mimics the process of morphological inflection. Our model generates words one character at a time by composing two latent representations: a continuous one, aimed at capturing the lexical semantics, and a set of (approximately) discrete features, aimed at capturing the morphosyntactic function, which are shared among different surface forms. Our model achieves better accuracy in translation into three morphologically-rich languages than conventional open-vocabulary NMT methods, while also demonstrating a better generalization capacity under low to mid-resource settings.

heading: 1 INTRODUCTION
text: Neural machine translation (NMT) models are conventionally trained by maximizing the likelihood of generating the target side of a bilingual parallel corpus of observations one word at a time conditioned of their full observed context. NMT models must therefore learn distributed representations that accurately predict word forms in very diverse contexts, a process that is highly demanding in terms of training data as well as the network capacity. Under conditions of lexical sparsity, which includes both the case of unknown words and the case of known words occurring in surprising contexts, the model is likely to struggle. Such adverse conditions are typical of translation involving morphologically-rich languages, where any single root may lead to exponentially many different surface realizations depending on its syntactic context. Such highly productive processes of word formation lead to many word forms being rarely or ever observed with a particular set of morphosyntactic attributes. The standard approach to overcome this limitation is to pre-process words into subword units that are shared among words, which are, in principle, more reliable as they are observed more frequently in varying context (Sennrich et al., 2016; Wu et al., 2016). One drawback related to this approach, however, is that the estimation of the subword vocabulary relies on word segmentation methods optimized using corpus-dependent statistics, disregarding any linguistic notion of morphology and the translation objective. This often produces subword units that are semantically ambiguous as they might be used in far too many lexical and syntactic contexts (Ataman et al., 2017). Moreover, in this approach, a word form is then generated by prediction of multiple subword units, which makes generalizing to unseen word forms more difficult due to the possibility that a subword unit necessary to reconstruct a given word form may be unlikely in a given context. To alleviate the sub-optimal effects of using explicit segmentation and generalize better to new morphological forms, recent studies explored the idea of extending NMT to model translation directly at
∗Work done while the first author was a doctoral student at the University of Trento and a visiting postgraduate student at the University of Edinburgh.
the level of characters (Kreutzer & Sokolov, 2018; Cherry et al., 2018), which, in turn, have demonstrated the requirement of using comparably deeper networks, as the network would then need to learn longer distance grammatical dependencies (Sennrich, 2017).
In this paper, we explore the benefits of explicitly modeling variation in surface forms of words using techniques from deep latent variable modeling in order to improve translation accuracy for low-resource and morphologically-rich languages. Latent variable models allow us to inject inductive biases relevant to the task, which, in our case, is word formation during translation. In order to formulate the process of morphological inflection, design a hierarchical latent variable model which translates words one character at a time based on word representations learned compositionally from sub-lexical components. In particular, for each word, our model generates two latent representations: i) a continuous-space dense vector aimed at capturing the lexical semantics of the word in a given context, and ii) a set of (approximately) discrete features aimed at capturing that word’s morphosyntatic role in the sentence. We then see inflection as decoding a word form, one character at a time, from a learned composition of these two representations. By forcing the model to encode each word representation in terms of a more compact set of latent features, we encourage them to be shared across contexts and word forms, thus, facilitating generalization under sparse settings. We evaluate our method in translating English into three morphologically-rich languages each with a distinct morphological typology: Arabic, Czech and Turkish, and show that our model is able to obtain better translation accuracy and generalization capacity than conventional approaches to open-vocabulary NMT.

heading: 2 NEURAL MACHINE TRANSLATION
text: In this paper, we use recurrent NMT architectures based on the model developed by Bahdanau et al. (2014). The model essentially estimates the conditional probability of translating a source sequence x = 〈x1, x2, . . . xm〉 into a target sequence y = 〈y1, y2, . . . yl〉 via an exact factorization:
p(y|x, θ) = l∏ i=1 p(yj |x, y<i, θ) (1)
where y<i stands for the sequence preceding the ith target word. At each step of the sequence, a fixed neural network architecture maps its inputs, the source sentence and the target prefix, to the probability of the ith target word observation in context. In order to condition on the source sentence fully, this network employs an embedding layer and a bi-directional recurrent neural network (bi-RNN) based encoder. Conditioning on the target prefix y<i is implemented using a recurrent neural network (RNN) based decoder, and an attention mechanism which summarises the source sentence into a context vector ci as a function of a given prefix (Luong et al., 2015). Given a parallel training set D, the parameters θ of the network are estimated to attain a local minimum of the negative log-likelihood function L(θ|D) = − ∑ x,y∼D log p(y|x, θ) via stochastic gradient-based optimization (Bottou & Cun, 2004).
Atomic parameterization estimates the probability of generating each target word yi in a single shot:
p(yi|x, y<i, θ) = exp ( E>yihi )∑v e=1 exp(E > e hi) , (2)
where E ∈ Rv×d is the target embedding matrix and the decoder output hi ∈ Rd represents x and y<i. Clearly, the size v of the target vocabulary plays an important role in determining the complexity of the model, which creates an important bottleneck when translating into low-resource and morphologically-rich languages due to the sparsity in the lexical distribution.
Recent studies approached this problem by performing NMT with subword units, a popular one of which is based on the Byte-Pair Encoding algorithm (BPE; Sennrich et al., 2016), which finds the optimal description of a corpus vocabulary by iteratively merging the most frequent character sequences. Atomic parameterization could also be used to model translation at the level of characters, which is found to be advantageous in generalizing to morphological variations (Cherry et al., 2018).
Hierarchical paramaterization further factorizes the probability of a target word in context:
p(yi|x, y<i, θ) = li∏ j=1 p(yi,j |x, y<i, yi,<j , θ) (3)
where the ith word yi = 〈yi,1, . . . , yi,li〉 is seen as a sequence of li characters. Generation follows one character at a time, each with probability computed by a fixed neural network architecture with varying inputs, namely, the source sentence x, the target prefix y<i, and the prefix yi,<j of characters already generated for that word. In this case there are two recurrent cells, one updated at the boundary of each token, much like in the standard case, and another updated at the character level. Luong & Manning (2016) propose hierarchical parameterization to compute the probability p(yi|x, y<i, θ) for unknown words, while for known words they use the atomic parameterization. In this paper, we use the hierarchical parameterization method for generating all target words, where we also augment the input embedding layer with a character-level bi-RNN, which computes each word representation yi as a composition of the embeddings of their characters (Ling et al., 2015).

heading: 3 A LATENT MORPHOLOGY MODEL (LMM) FOR LEARNING WORD REPRESENTATIONS
text: The application of a hierarchical structure for learning word representations in language modeling (Vania & Lopez, 2017) or semantic role labeling (Sahin & Steedman, 2018) have shown that such representations encode many cues about the morphological features of words by establishing a mapping between phonetic units and lexical context. Although it can provide an alternative solution to open-vocabulary NMT by potentially alleviating the need for subword segmentation, the quality of word representations learned by an hierarchical model is still highly dependant on the amount of observations (Sahin & Steedman, 2018; Ataman et al., 2019), since the training data is essential in properly modeling the lexical distribution. On the other hand, the process of word formation, particularly morphological inflection, has many properties that remain universal across languages, where a word is typically composed of a lemma, representing its lexical semantics, and a distinct combination of categorical inflectional features expressing the word’s syntactic role in the phrase or sentence. In this paper, we propose to manipulate this universal structure in order to enforce an inductive bias on the prior distribution of words and allow the hierarchical parameterization model in properly learning lexical representations under conditions of data sparsity.

heading: 3.1 GENERATIVE MODEL
text: Our generative LMM for NMT formulates word formation in terms of a stochastic process, where each word is generated one character at a time by composing two latent representations: a continuous vector aimed at capturing the lemma, and a set of sparse features aimed at capturing the inflectional features. The motivation for using a stochastic model is twofold. First, deterministic models are by definition unimodal: when presented with the same input (the same context) they always produce the same output. When we model the word formation process, it is reasonable to expect a larger degree of ambiguity, that is, for the same context (e.g. a noun prefix), we may continue by inflecting the word differently depending on the (latent) mode of operation we are at (e.g. generating nominative, accusative or dative noun). Second, in stochastic models, the choice of distribution gives us a mechanism to favour a particular type of representation. In our case, we use sparse distributions for inflectional features to accommodate the fact that morphosyntactic features are discrete in nature. Our latent variable model is an instance of a variational auto-encoder (VAE; Kingma & Welling, 2013) inspired by the model of Zhou & Neubig (2017) for morphological reinflection.
Generation of the ith word starts by sampling a Gaussian-distributed representation in context. This requires predicting the Gaussian location ui and scale si vectors,1
Zi|x, y<i ∼ N (ui,diag(si si)) ui = dense(hi; θu)
si = ζ(dense(hi; θs))
(4)
1Notation We use capital Roman letters for random variables (and lowercase letters for assignments). Boldface Roman letters are reserved for neural network output vectors, and stands for elementwise multiplication. Finally, we denote typical neural network layers as layer(inputs; parameters).
where prediction of the location (in Rd) and scale (in Rd>0) from the word-level decoder hidden state hi (which represents x and y<i) is performed by two dense layers, and the scale values are ensured to be positive with the softplus (ζ) activation.2
Generation proceeds by then sampling aK-dimensional vector fi of sparse scalar features (see §3.2) conditioned on the source x, the target prefix y<i, and the sampled lemma zi. We model sampling of fi conditioned on zi in order to capture the insight that inflectional transformations typically depend on the category of a lemma. Having sampled fi and zi, the representation of the ith target word is computed by a transformation of zi and fi, i.e. ti = dense([zi, fi]; θcomp).
As shown in Figure 1, our model generates each word character by character auto-regressively by conditioning on the word representation ti predicted by the LMM, the current context ci, and the previously generated characters following the hierarchical parameterization.3 See Algorithm 1 for details on generation.
Input: model parameters θ, latent lemma zi, latent morphological attributes fi, observed character sequence 〈yi,1, . . . , yi,li〉 if training or placeholders if test, decoder state hi, and context vector ci Result: updated decoder hidden state, prediction (a word), probability of prediction (for loss) initialization; ti = dense([zi, fi]; θcomp) ; initialize char-rnn with a projection of [ti, ci]; for j < li and j < max do
compute output layer from char-rnn state ; if training then
set prediction to observation yi,j ; else
set prediction to argmax of output softmax layer ; end assess log-probability of prediction ; update word-level RNN decoder with prediction;
end Algorithm 1: Word generation: in training the word is observed, thus we only update the decoder and assess the probability of the observation, in test, we use mean values of the distributions to represent most likely values for z and f and populate predictions with beam-search.
2In practice, we sample zi via a reparameterization in terms of a fixed Gaussian, namely, zi = ui + i si for i ∼ N (0, Id). This is known as the reparameterization trick (Kingma & Welling, 2013), which allows back-propagation through stochastic units (Rezende et al., 2014).
3Formally, because the decoder is an RNN, we are also conditioning on z<i and f<i. We omit this dependence to avoid clutter.

heading: 3.2 SPARSE FEATURES
text: Since each target word yi may have multiple inflectional features, ideally, we would like fi to be K feature indicators, which could be achieved by sampling from K independent Bernoulli distributions parameterized in context. The problem with this approach is that sampling Bernoulli outcomes is non-differentiable, thus, their training requires gradient estimation via REINFORCE (Williams, 1992) and sophisticated variance reduction techniques. An alternative approach that has recently become popular is to use relaxations such as the Concrete distribution or Gumbel-Softmax (Maddison et al., 2017; Jang et al., 2017) in combination with the straight-through estimator (ST; Bengio et al., 2013). This is based on the idea of relaxing the discrete variable from taking on samples in the discrete set {0, 1} to taking on samples in the continuous set (0, 1) using a distribution for which a reparameterization exists (e.g. Gumbel). Then, a non-differentiable activation (e.g. a threshold function) maps continuous outcomes to discrete ones. ST simply ignores the discontinuous activation in the backward pass, i.e. it assumes the Jacobian is the identity matrix. This does lead to biased estimates of the gradient of the loss, which is in conflict with the requirements behind stochastic optimization (Robbins & Monro, 1951).
An alternative presented by Louizos et al. (2018) achieves a different compromise, it gets rid of bias at the cost of mixing both sparse and dense outcomes. The idea is to obtain a continuous sample c ∈ (0, 1) from a distribution for which a reparameterization exists and stretch it to a continuous support (l, r) ⊃ (0, 1) using a simple linear transformation s = l + (r − l)c. A rectifier is then employed to map the negative outcomes to 0 and the positive outcomes larger than one to 1, i.e. f = min(1,max(0, s)). The rectifier is only non-differentiable at s = 0 and at s = 1, however, because the stretched variable s is sampled from a continuous distribution, the chance of sampling s = 0 and s = 1 is essentially 0. This stretched-and-rectified distribution allows: i) the sampling procedure to become differentiable with respect to the parameters of the distribution, ii) to sample sparse outcomes with an unbiased estimator, and iii) to calculate the probability of sampling f = 0 and f = 1 in closed form as a function of the parameters of the underlying distribution, which corresponds to the probability of sampling s < 0 and s > 1, respectively.
In their paper, Louizos et al. (2018) used the BinaryConcrete (or Gumbel-Sigmoid) as the underlying continuous distribution, the sparsity of which is controlled via a temperature parameter. However, in our study, we found this parameter difficult to predict, since it is very hard to allow a neural network to control its value without unstable gradient updates. Instead, we opt for a slight variant by Bastings et al. (2019) based on the Kumaraswamy distribution (Kumaraswamy, 1980), a twoparameters distribution that closely resembles a Beta distribution and is sparse whenever its (strictly positive) parameters are between 0 and 1. In the context of text classification, Bastings et al. (2019) shows this stretch-and-rectify technique to work better than methods based on REINFORCE.
For each token yi, we sample K independent Kumaraswamy variables in context, Ci,k|x, y<i, zi ∼ Kuma(ai,k, bi,k) k = 1, . . . ,K [ai,bi] = ζ(dense([zi,hi]; θab))
(5)
which makes a continuous random vector ci in the support (0, 1)K .4 We then stretch-and-rectify the samples via fi,k = min(1,max(0, l − (r − l)ci,k)) making fi a random vector in the support [0, 1]K .5 The probability that fi,k is exactly 0 is
π {0} i,k =
∫ −l r−l
0
Kuma(c|ai,k, bi,k)dc (6a)
and the probability that fi,k is exactly 1 is
π {1} i,k = 1−
∫ 1−l r−l
0
Kuma(c|ai,k, bi,k)dc (6b)
and therefore the complement π (0,1) i,k = 1− π {0} i,k − π {1} i,k (6c)
4In practice we sample ci,k via a reparameterization of a fixed uniform variable, namely, ci,k = (1 − (1 − εi,k)1/bi,k )1/ai,k where εi,k ∼ U(0, 1), which much like the Gaussian reparameterization enables backpropagation through samples (Nalisnick & Smyth, 2016).
5We use l = −0.1 and r = 1.1. Figure 2 in the appendix illustrates different instances of this distribution.
is the probability that fi,k be any continuous value in the open set (0, 1). In §3.4, we will derive regularizers based on π(0,1)i,k to promote sparse outcomes to be sampled with large probability.

heading: 3.3 PARAMETER ESTIMATION
text: Parameter estimation of neural network models is typically done via maximum-likelihood estimation (MLE), where we approach a local minimum of the negative log-likelihood function via stochastic gradient descent with gradient computation automated by the back-propagation algorithm. Using the following shorthand notation:
α(zi) , p(zi|x, y<i, z<i, f<i, θ) (7a)
β(fi) , K∏ k=1 p(fi,k|x, y<i, z<i, f<i, zi, θ) (7b)
γ(yi) , li∏ j=1 p(yi,j |x, y<i, z≤i, f≤i, yi,<j , θ) . (7c)
The log-likelihood for a single data point can be formulated as: log p(y|x, θ) = log ∫ l∏
i=1
α(zi)β(fi)γ(yi)dzdf (8)
the computation of which is intractable. Instead, we resort to variational inference (VI; Jordan et al., 1999), where we optimize a lower-bound on the log-likelihood
Eq(z,f |x,y,λ) [ l∑ i=1 log α(zi)β(fi)γ(yi) q(z, f |x, λ) ] (9)
expressed with respect to an independently parameterized posterior approximation q(z, f |x, y, λ). For as long as sampling from the posterior is tractable and can be performed via a reparameterization, we can rely on stochastic gradient-based optimization. In order to have a compact parameterization, we choose
q(z, f |x, y, λ) := l∏ i=1 α(zi)β(fi) . (10)
This simplifies the lowerbound, which then takes the form of l nested expectations, the ith of which is Eα(zi)β(fi) [log γ(yi)]. This is similar to the stochastic decoder of Schulz et al. (2018), though our approximate posterior is in fact, also our parameterized prior.6 Although this objective does not particularly promote sparsity, we employ sparsity-inducing regularization techniques that will be discussed in the next section.
Concretely, for a given source sentence x, target prefix y<i, and a latent sample z≤i, f≤i, we obtain a single-sample estimate of the loss by computing Li(θ) = − log γ(yi).

heading: 3.4 REGULARIZATION
text: In order to promote sparse distributions for the inflectional features, we apply a regularizer inspired by expected L0 regularization (Louizos et al., 2018). Whereas L0 is a penalty based on the number of non-zero outcomes, we design a penalty based on the expected number of continuous outcomes, which corresponds to π(0,1)i,k as shown in Equation (6). For a given source sentence x, target prefix y<i, and a latent sample z<i, f<i, we aggregate this penalty for each feature
Ri(θ) = K∑ k=1 π (0,1) i,k (11)
6This means we refrain from conditioning on the observation yi itself when sampling zi and fi. Whereas this gives our posterior approximation access to less features than the true posterior would have, we do not employ a fixed uninformative prior, but rather an autoregressive network trained on the likelihood of generated word forms. This is a common approximation for latent variable models that employ autoregressive priors (Goyal et al., 2017).
and add it to the cost function with a positive weight ρ. The final loss of the NMT model is
L(θ|D) = ∑ x,y∼D |y|∑ i=1 Li(θ) + ρRi(θ) . (12)

heading: 3.5 PREDICTIONS
text: In our model, obtaining the conditional likelihood for predicting the most likely hypothesis requires marginalisation of the latent variables, which is intractable. An alternative approach is to heuristically search through the joint distribution,
argmax y,z,f
p(y, z, f |x) , (13)
rather than the marginal, an approximation that has been referred to as Viterbi decoding (Smith, 2011). During beam search, we populate the beam with alternative target words, and for each prefix y<i in the beam, we resort to deterministically choosing the latent variables based on a single sample which we deem representative of their distributions, which is a common heuristic in VAEs for translation (Zhang et al., 2016; Schulz et al., 2018). For unimodal distributions, such as the Gaussian p(zi|x, y<i, z<i, f<i), we use the analytic mean, whereas for multimodal distributions, such as the Hard Kumaraswamy p(fi|x, y<i, z≤i, f<i), we use the argmax.7

heading: 4 EVALUATION

heading: 4.1 MODELS
text: We evaluate our model by comparing it in machine translation against three baselines which constitute the conventional open-vocabulary NMT methods, including architectures using atomic parameterization either with subword units segmented with BPE (Sennrich et al., 2016) or characters, and the hierarchical parameterization method employed for generating all words in the output. We implement all architectures using Pytorch (Paszke et al., 2017) within the OpenNMT-py framework (Klein et al., 2017)8.

heading: 4.2 DATA AND LANGUAGES
text: In order to evaluate our model we design two sets of experiments. The experiments in §4.4.1 aim to evaluate different methods under low-resource settings, for languages with different morphological typology. We model the machine translation task from English into three languages with distinct morphological characteristics: Arabic (templatic), Czech (fusional), and Turkish (agglutinative). We use the TED Talks corpora (Cettolo, 2012) for training the NMT models for these experiments. In §4.4.3, we conduct more experiments in Turkish to demonstrate the case of increased data sparsity using multi-domain training corpora, where we extend the training set using corpora from EU Bookshop (Skadiņš et al., 2014), Global Voices, Gnome, Tatoeba, Ubuntu (Tiedemann, 2012), KDE4 (Tiedemann, 2009), Open Subtitles (Lison & Tiedemann, 2016) and SETIMES (Tyers & Alperen, 2010)9. The statistical characteristics of the training sets are given in Tables 4 and 5. We use the official evaluation sets of the IWSLT10 for validating and testing the accuracy of the models. In order to increase the number of unknown and rare words in the evaluation sets we measure accuracy on large test sets combining evaluation sets from many years (Table 6 presents the evaluation sets used for development and testing). The accuracy of each model output is measured using BLEU (Papineni et al., 2002) and chrF3 (Popović, 2015) metrics, whereas the significance of the improvements are computed using bootstrap hypothesis testing (Clark et al., 2011). In order to measure the accuracy in predicting the correct syntactic description of the references, we also compute BLEU
7We maximize across the three configurations of each feature, namely, max{π{0}i,k , π {1} i,k , π (0,1) i,k }. If π (0,1) i,k is highest, we return the mean of the underlying Kumaraswamy variable. 8Our software is available at: https://github.com/d-ataman/lmm 9The size of the resulting combined corpora is further reduced to filter out noise and reduce the computational cost of the experiments using data selection methods (Cuong & Simaan, 2014). 10The International Workshop on Spoken Language Translation
scores over the output sentences segmented using a morphological analyzer. We use the AlKhalil Morphosys (Boudchiche et al., 2017) for segmenting Arabic, Morphidata (Straková et al., 2014) for segmenting Czech and the morphological lexicon model of Oflazer (Oflazer & Kuruöz, 1994) and disambiguation tool of Sak (Sak et al., 2007) for segmenting Turkish sentences into sequences of lemmas and morphological features.

heading: 4.3 TRAINING SETTINGS
text: All models are implemented using gated recurrent units (GRU) (Cho et al., 2014), and have a singlelayer bi-RNN encoder. The source sides of the data used for training all NMT models, and the target sides of the data used in training the subword-level NMT models are segmented using BPE with 16,000 merge rules. We implement all decoders using a comparable number of GRU parameters, including 3-layer stacked-GRU subword and character-level decoders, where the attention is computed after the 1st layer (Barone et al., 2017) and a 3-layer hierarchical decoder which implements the attention mechanism after the 2nd layer. All models use an embedding dimension and GRU size of 512. LMM uses the same hierarchical GRU architecture, where the middle layer is augmented using 4 multi-layer perceptrons with 256 hidden units. We use a lemma vector dimension of 150, 10 inflectional features (See §A.3 for experiments conducted to tune the feature dimensions) and set the regularization constant to ρ = 0.4. All models are trained using the Adam optimizer (Kinga & Ba, 2014) with a batch size of 100, dropout rate of 0.2, learning rate of 0.0004 and learning rate decay of 0.8, applied when the perplexity does not decrease at a given epoch.11 Translations are generated with beam search with a beam size of 5, where the hierarchical models implement the hierarchical beam search algorithm (Ataman et al., 2019).

heading: 4.4 RESULTS

heading: 4.4.1 THE EFFECT OF MORPHOLOGICAL TYPOLOGY
text: The experiment results given in Table 1 show the performance of each model in translating English into Arabic, Czech and Turkish. In Turkish, the most sparse target language in our benchmark with rich agglutinative morphology, using character-based decoding shows to be more advantageous compared to the subword-level and hierarchical models, suggesting that increased granularity in the vocabulary units might aid in better learning accurate representations under conditions of high data sparsity. In Arabic, on the other hand, using a hierarchical decoding model shows to be advantageous compared to the subword and character-level models, as it might be useful in better learning syntactic dependencies. LMM obtains improvements of 0.51 and 0.30 BLEU points in Arabic and Turkish over the best performing baselines, respectively. The fact that our model can efficiently work in both Arabic and Turkish confirms that it can handle the generation of both concatenative and non-concatenative morphological transformations. The results in the English-to-Czech translation direction do not indicate a specific advantage of using either method for generating fusional morphology, where morphemes are already optimized at the surface level, although our model is still able to achieve translation accuracy comparable to the character and subword-level models.

heading: 4.4.2 PREDICTING UNSEEN WORDS
text: In addition to the general machine translation evaluation using automatic metrics, we perform a more focused statistical analysis to illustrate the performance of different methods in predicting unseen words by computing the average perplexity per character on the input sentences which contain outof-vocabulary (OOV) words as suggested by Cotterell et al. (2018). We also analyze the outputs generated by each decoder in terms of the frequency of unknown words in each model output and the Kullback-Leibler (KL) divergence between the character trigram distributions of the references and outputs, which represents the coherence between the statistical distribution learned by each model and the reference translations.
Our analysis results generally confirm the advantage of increased granularity during the generation of unseen words, where the character-level decoder can generate a higher rate of unseen word forms and higher KL-divergence with the reference, suggesting superior ability in generalizing to new
11Perplexity is the exponentiated average negative log-likelihood per segment (BPE, or character) that a model assigns to a dataset. It corresponds to the model’s average surprisal per time step.
output and not necessarily copying previous observations as the subword-level model, however, this advantage is more visible in Turkish and less in Czech or Arabic. The hierarchical decoder which performs the search at the level of words, on the other hand, behaves with less uncertainty in terms of the perplexity values although it cannot demonstrate the ability to generalize to new forms and neither can closely capture the actual distribution in the target language.
Due to its stochastic nature, our model yields higher perplexity values compared to the hierarchical model, whereas the values range between subword and character-based models, possibly finding an optimal level of granularity between the two solutions. The KL-divergence and OOV rates confirm that our model has the potential in better generalize to new word forms as well as different morphological typology.

heading: 4.4.3 THE EFFECT OF DATA SIZE
text: Repeating the experiments in the English-to-Turkish translation direction by increasing the amount of training data with multi-domain corpora demonstrates a more challenging case, where there is a greater possibility of observing new words in varying context, either in the form of morphological inflections due to previously unobserved syntactic conditions, or a larger vocabulary extended with terminology from different domains. In this experiment, the character-level model experiences a drop in performance and its accuracy is much lower than the subword-level one, suggesting that its capacity cannot cope with the increased amount of training data. Empirical results suggest that with increased capacity, character-level models carry the potential to reach comparable performance to subword-level models (Cherry et al., 2018). On the other hand, our model reaches a much larger improvement of 0.82 BLEU points over the subword-level and 2.54 BLEU points over the character-level decoders, suggesting that it could make use of the increased amount of observations for improving the translation performance, which possibly aid the morphology model in becoming more accurate.

heading: 4.4.4 THE IMPACT OF INFLECTIONAL FEATURES
text: In order to understand whether the latent inflectional features in fact capture information about variations related to morphological transformations, we first try generating different surface forms of the same lemma by sampling a lemma vector with LMM for the input word ’go’ and generating outputs using the fixed lemma vector and assigning different values to the inflectional features. In the second experiment, we assess the impact of the inflectional features by setting all features f to 0 and translating a set of English sentences with varying inflected forms in Turkish. Table 3 presents different sets of feature values and the corresponding outputs generated by the decoder and the outputs generated with or without the inflectional component.
The model generates different surface forms for different sets of features, confirming that the latent variables represent morphological features related to the infinitive form of the verb, as well as its formality conditions, prepositions, person, number and tense. Decoding the set of sentences given in the second experiment LMM always generates the correct inflectional form, although when the feature values are set to 0 the model omits some inflectional features in the output, suggesting that despite partially relying on the source-side context, it still encodes important information for generating correct surface forms in the inflectional features.

heading: 5 CONCLUSION
text: In this paper we presented a novel decoding architecture for NMT employing a hierarchical latent variable model to promote sparsity in lexical representations, which demonstrated promising application for morphologically-rich and low-resource languages. Our model generates words one character at a time by composing two latent features representing their lemmas and inflectional features. We evaluate our model against conventional open-vocabulary NMT solutions such as subword and character-level decoding methods in translationg English into three morphologically-rich languages with different morphological typologies under low to mid-resource settings. Our results show that our model can significantly outperform subword-level NMT models, whereas demonstrates better capacity than character-level models in coping with increased amounts of data sparsity. We also conduct ablation studies on the impact of feature variations to the predictions, which prove that despite being completely unsupervised, our model can in fact manage to learn morphosyntactic information and make use of it to generalize to different surface forms of words.

heading: 6 ACKNOWLEDGMENTS
text: The authors would like to thank Marcello Federico, Orhan Fırat, Adam Lopez, Graham Neubig, Akash Srivastava and Clara Vania for their feedback and suggestions. This project received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements 825299 (GoURMET) and 688139 (SUMMA).

heading: A APPENDIX
text: A.1 THE STATISTICAL CHARACTERISTICS OF EXPERIMENTAL DATA
A.2 THE KUMARASWAMY DISTRIBUTION
A.3 THE EFFECT OF FEATURE DIMENSIONS
We investigate the optimal lemma and inflectional feature sizes by measuring the accuracy in English-to-Turkish translation using different feature vector dimensions. The results given in Figure 3 show that gradually compressing the word representations computed by recurrent hidden states, with an original dimension of 512, from 500 to 100, leads to increased output accuracy, suggesting that encoding more compact representations might provide the model with a better generalization capability. Our results also show that using a feature dimension of 10 is sufficient in reaching the best accuracy.

Review summaries:
This paper proposes a method, Latent Morphology Model (LMM), for producing word representations for a (hierarchical) character-level decoder used in neural machine translation (NMT). The main motivation is to overcome vocabulary sparsity or highly inflectional languages such as Arabic, Czech, and Turkish (experimented in the paper). To model the morphological inflection, they decouple lemmas and inflection types into 2 latent variables (z and f) where f is enforced to be sparse (arguably mimic the process of the human). The literature review of NMT and the discussion on the potential advantage of morphology are concise. The proposed model is a variation of Luong & Manning (2016) and Schulz et al (2018) models, thus, their main contribution is an introduction of the latent morphological features to the decoder. The proposed LMM is trained by sampling z and f from prior directly, and the sparsity of the morphological features is encouraged by L0 of the feature vector (parameterized as independent Kumaraswamy variables). They perform the main empirical study using 3 languages by translating from English to justify the proposed LMM. Lastly, they provide a quantitative analysis of the perplexities of unseen words and a qualitative on words generated of a lemma with different feature vectors.