Title: A LATENT MORPHOLOGY MODEL FOR OPEN-VOCABULARY NEURAL MACHINE TRANSLATION

Abstract: Translation into morphologically-rich languages challenges neural machine translation (NMT) models with extremely sparse vocabularies where atomic treatment of surface forms is unrealistic. This problem is typically addressed by either pre-processing words into subword units or performing translation directly at the level of characters. The former is based on word segmentation algorithms optimized using corpus-level statistics with no regard to the translation task. The latter learns directly from translation data but requires rather deep architectures. In this paper, we propose to translate words by modeling word formation through a hierarchical latent variable model which mimics the process of morphological inflection. Our model generates words one character at a time by composing two latent representations: a continuous one, aimed at capturing the lexical semantics, and a set of (approximately) discrete features, aimed at capturing the morphosyntactic function, which are shared among different surface forms. Our model achieves better accuracy in translation into three morphologically-rich languages than conventional open-vocabulary NMT methods, while also demonstrating a better generalization capacity under low to mid-resource settings.

1. INTRODUCTION:

Neural machine translation (NMT) models are conventionally trained by maximizing the likelihood of generating the target side of a bilingual parallel corpus of observations one word at a time conditioned of their full observed context. NMT models must therefore learn distributed representations that accurately predict word forms in very diverse contexts, a process that is highly demanding in terms of training data as well as the network capacity. Under conditions of lexical sparsity, which includes both the case of unknown words and the case of known words occurring in surprising contexts, the model is likely to struggle. Such adverse conditions are typical of translation involving morphologically-rich languages, where any single root may lead to exponentially many different surface realizations depending on its syntactic context. Such highly productive processes of word formation lead to many word forms being rarely or ever observed with a particular set of morphosyntactic attributes. The standard approach to overcome this limitation is to pre-process words into subword units that are shared among words, which are, in principle, more reliable as they are observed more frequently in varying context (Sennrich et al., 2016; Wu et al., 2016) . One drawback related to this approach, however, is that the estimation of the subword vocabulary relies on word segmentation methods optimized using corpus-dependent statistics, disregarding any linguistic notion of morphology and the translation objective. This often produces subword units that are semantically ambiguous as they might be used in far too many lexical and syntactic contexts (Ataman et al., 2017) . Moreover, in this approach, a word form is then generated by prediction of multiple subword units, which makes generalizing to unseen word forms more difficult due to the possibility that a subword unit necessary to reconstruct a given word form may be unlikely in a given context. To alleviate the sub-optimal effects of using explicit segmentation and generalize better to new morphological forms, recent studies explored the idea of extending NMT to model translation directly at the level of characters (Kreutzer & Sokolov, 2018; Cherry et al., 2018) , which, in turn, have demonstrated the requirement of using comparably deeper networks, as the network would then need to learn longer distance grammatical dependencies (Sennrich, 2017) .

In this paper, we explore the benefits of explicitly modeling variation in surface forms of words using techniques from deep latent variable modeling in order to improve translation accuracy for low-resource and morphologically-rich languages. Latent variable models allow us to inject inductive biases relevant to the task, which, in our case, is word formation during translation. In order to formulate the process of morphological inflection, design a hierarchical latent variable model which translates words one character at a time based on word representations learned compositionally from sub-lexical components. In particular, for each word, our model generates two latent representations: i) a continuous-space dense vector aimed at capturing the lexical semantics of the word in a given context, and ii) a set of (approximately) discrete features aimed at capturing that word's morphosyntatic role in the sentence. We then see inflection as decoding a word form, one character at a time, from a learned composition of these two representations. By forcing the model to encode each word representation in terms of a more compact set of latent features, we encourage them to be shared across contexts and word forms, thus, facilitating generalization under sparse settings. We evaluate our method in translating English into three morphologically-rich languages each with a distinct morphological typology: Arabic, Czech and Turkish, and show that our model is able to obtain better translation accuracy and generalization capacity than conventional approaches to open-vocabulary NMT.

2. NEURAL MACHINE TRANSLATION:

In this paper, we use recurrent NMT architectures based on the model developed by Bahdanau et al. (2014) . The model essentially estimates the conditional probability of translating a source sequence x = x 1 , x 2 , . . . x m into a target sequence y = y 1 , y 2 , . . . y l via an exact factorization:

EQUATION

where y <i stands for the sequence preceding the ith target word. At each step of the sequence, a fixed neural network architecture maps its inputs, the source sentence and the target prefix, to the probability of the ith target word observation in context. In order to condition on the source sentence fully, this network employs an embedding layer and a bi-directional recurrent neural network (bi-RNN) based encoder. Conditioning on the target prefix y <i is implemented using a recurrent neural network (RNN) based decoder, and an attention mechanism which summarises the source sentence into a context vector c i as a function of a given prefix (Luong et al., 2015) . Given a parallel training set D, the parameters θ of the network are estimated to attain a local minimum of the negative log-likelihood function L(θ|D) =x,y∼D log p(y|x, θ) via stochastic gradient-based optimization (Bottou & Cun, 2004) .

Atomic parameterization estimates the probability of generating each target word y i in a single shot:

EQUATION

where E ∈ R v×d is the target embedding matrix and the decoder output h i ∈ R d represents x and y <i . Clearly, the size v of the target vocabulary plays an important role in determining the complexity of the model, which creates an important bottleneck when translating into low-resource and morphologically-rich languages due to the sparsity in the lexical distribution.

Recent studies approached this problem by performing NMT with subword units, a popular one of which is based on the Byte-Pair Encoding algorithm (BPE; Sennrich et al., 2016) , which finds the optimal description of a corpus vocabulary by iteratively merging the most frequent character sequences. Atomic parameterization could also be used to model translation at the level of characters, which is found to be advantageous in generalizing to morphological variations (Cherry et al., 2018) .

Hierarchical paramaterization further factorizes the probability of a target word in context:

EQUATION

where the ith word y i = y i,1 , . . . , y i,li is seen as a sequence of l i characters. Generation follows one character at a time, each with probability computed by a fixed neural network architecture with varying inputs, namely, the source sentence x, the target prefix y <i , and the prefix y i,<j of characters already generated for that word. In this case there are two recurrent cells, one updated at the boundary of each token, much like in the standard case, and another updated at the character level. Luong & Manning (2016) propose hierarchical parameterization to compute the probability p(y i |x, y <i , θ) for unknown words, while for known words they use the atomic parameterization. In this paper, we use the hierarchical parameterization method for generating all target words, where we also augment the input embedding layer with a character-level bi-RNN, which computes each word representation y i as a composition of the embeddings of their characters (Ling et al., 2015) .

3. A LATENT MORPHOLOGY MODEL (LMM) FOR LEARNING WORD REPRESENTATIONS:

The application of a hierarchical structure for learning word representations in language modeling (Vania & Lopez, 2017) or semantic role labeling (Sahin & Steedman, 2018) have shown that such representations encode many cues about the morphological features of words by establishing a mapping between phonetic units and lexical context. Although it can provide an alternative solution to open-vocabulary NMT by potentially alleviating the need for subword segmentation, the quality of word representations learned by an hierarchical model is still highly dependant on the amount of observations (Sahin & Steedman, 2018; Ataman et al., 2019) , since the training data is essential in properly modeling the lexical distribution. On the other hand, the process of word formation, particularly morphological inflection, has many properties that remain universal across languages, where a word is typically composed of a lemma, representing its lexical semantics, and a distinct combination of categorical inflectional features expressing the word's syntactic role in the phrase or sentence. In this paper, we propose to manipulate this universal structure in order to enforce an inductive bias on the prior distribution of words and allow the hierarchical parameterization model in properly learning lexical representations under conditions of data sparsity.

3.1. GENERATIVE MODEL:

Our generative LMM for NMT formulates word formation in terms of a stochastic process, where each word is generated one character at a time by composing two latent representations: a continuous vector aimed at capturing the lemma, and a set of sparse features aimed at capturing the inflectional features. The motivation for using a stochastic model is twofold. First, deterministic models are by definition unimodal: when presented with the same input (the same context) they always produce the same output. When we model the word formation process, it is reasonable to expect a larger degree of ambiguity, that is, for the same context (e.g. a noun prefix), we may continue by inflecting the word differently depending on the (latent) mode of operation we are at (e.g. generating nominative, accusative or dative noun). Second, in stochastic models, the choice of distribution gives us a mechanism to favour a particular type of representation. In our case, we use sparse distributions for inflectional features to accommodate the fact that morphosyntactic features are discrete in nature.

Our latent variable model is an instance of a variational auto-encoder (VAE; Kingma & Welling, 2013) inspired by the model of Zhou & Neubig (2017) for morphological reinflection.

Generation of the ith word starts by sampling a Gaussian-distributed representation in context. This requires predicting the Gaussian location u i and scale

EQUATION

Figure 1 : LMM for computing word representations while translating the sentence '... went home' into Turkish ('eve-(to)home gitti(he/she/it)went'). The character-level decoder is initialized with the attentional vector h i computed by the attention mechanism using current context c i and the word representation t i as in Luong & Manning (2016) .

where prediction of the location (in R d ) and scale (in R d >0 ) from the word-level decoder hidden state h i (which represents x and y <i ) is performed by two dense layers, and the scale values are ensured to be positive with the softplus (ζ) activation. 2Generation proceeds by then sampling a K-dimensional vector f i of sparse scalar features (see §3.2) conditioned on the source x, the target prefix y <i , and the sampled lemma z i . We model sampling of f i conditioned on z i in order to capture the insight that inflectional transformations typically depend on the category of a lemma. Having sampled f i and z i , the representation of the ith target word is computed by a transformation of z i and f i , i.e.

t i = dense([z i , f i ]; θ comp ).

As shown in Figure 1 , our model generates each word character by character auto-regressively by conditioning on the word representation t i predicted by the LMM, the current context c i , and the previously generated characters following the hierarchical parameterization. 3 See Algorithm 1 for details on generation.

Input: model parameters θ, latent lemma z i , latent morphological attributes f i , observed character sequence y i,1 , . . . , y i,li if training or placeholders if test, decoder state h i , and context vector c i Result: updated decoder hidden state, prediction (a word), probability of prediction (for loss) initialization;

t i = dense([z i , f i ]; θ comp ) ; initialize char-rnn with a projection of [t i , c i ];

for j < l i and j < max do compute output layer from char-rnn state ;

if training then set prediction to observation y i,j ; else set prediction to arg max of output softmax layer ; end assess log-probability of prediction ; update word-level RNN decoder with prediction; end Algorithm 1: Word generation: in training the word is observed, thus we only update the decoder and assess the probability of the observation, in test, we use mean values of the distributions to represent most likely values for z and f and populate predictions with beam-search.

3.2. SPARSE FEATURES:

Since each target word y i may have multiple inflectional features, ideally, we would like f i to be K feature indicators, which could be achieved by sampling from K independent Bernoulli distributions parameterized in context. The problem with this approach is that sampling Bernoulli outcomes is non-differentiable, thus, their training requires gradient estimation via REINFORCE (Williams, 1992) and sophisticated variance reduction techniques. An alternative approach that has recently become popular is to use relaxations such as the Concrete distribution or Gumbel-Softmax (Maddison et al., 2017; Jang et al., 2017) in combination with the straight-through estimator (ST; Bengio et al., 2013) . This is based on the idea of relaxing the discrete variable from taking on samples in the discrete set {0, 1} to taking on samples in the continuous set (0, 1) using a distribution for which a reparameterization exists (e.g. Gumbel). Then, a non-differentiable activation (e.g. a threshold function) maps continuous outcomes to discrete ones. ST simply ignores the discontinuous activation in the backward pass, i.e. it assumes the Jacobian is the identity matrix. This does lead to biased estimates of the gradient of the loss, which is in conflict with the requirements behind stochastic optimization (Robbins & Monro, 1951 ).

An alternative presented by Louizos et al. (2018) achieves a different compromise, it gets rid of bias at the cost of mixing both sparse and dense outcomes. The idea is to obtain a continuous sample c ∈ (0, 1) from a distribution for which a reparameterization exists and stretch it to a continuous support (l, r) ⊃ (0, 1) using a simple linear transformation s = l + (r -l)c. A rectifier is then employed to map the negative outcomes to 0 and the positive outcomes larger than one to 1, i.e. f = min(1, max(0, s)). The rectifier is only non-differentiable at s = 0 and at s = 1, however, because the stretched variable s is sampled from a continuous distribution, the chance of sampling s = 0 and s = 1 is essentially 0. This stretched-and-rectified distribution allows: i) the sampling procedure to become differentiable with respect to the parameters of the distribution, ii) to sample sparse outcomes with an unbiased estimator, and iii) to calculate the probability of sampling f = 0 and f = 1 in closed form as a function of the parameters of the underlying distribution, which corresponds to the probability of sampling s < 0 and s > 1, respectively.

In their paper, Louizos et al. (2018) used the BinaryConcrete (or Gumbel-Sigmoid) as the underlying continuous distribution, the sparsity of which is controlled via a temperature parameter. However, in our study, we found this parameter difficult to predict, since it is very hard to allow a neural network to control its value without unstable gradient updates. Instead, we opt for a slight variant by Bastings et al. (2019) based on the Kumaraswamy distribution (Kumaraswamy, 1980) , a twoparameters distribution that closely resembles a Beta distribution and is sparse whenever its (strictly positive) parameters are between 0 and 1. In the context of text classification, Bastings et al. (2019) shows this stretch-and-rectify technique to work better than methods based on REINFORCE.

For each token y i , we sample K independent Kumaraswamy variables in context,

EQUATION

which makes a continuous random vector c i in the support (0, 1) K . 4 We then stretch-and-rectify the samples via

f i,k = min(1, max(0, l -(r -l)c i,k )) making f i a random vector in the support [0, 1] K . 5 The probability that f i,k is exactly 0 is π {0} i,k = -l r-l 0 Kuma(c|a i,k , b i,k )dc (6a)

and the probability that f i,k is exactly 1 is

π {1} i,k = 1 - 1-l r-l 0 Kuma(c|a i,k , b i,k )dc (6b)

and therefore the complement π

EQUATION

4 In practice we sample c i,k via a reparameterization of a fixed uniform variable, namely,

c i,k = (1 - (1 -ε i,k ) 1/b i,k )

1/a i,k where ε i,k ∼ U(0, 1), which much like the Gaussian reparameterization enables backpropagation through samples (Nalisnick & Smyth, 2016) . is the probability that f i,k be any continuous value in the open set (0, 1). In §3.4, we will derive regularizers based on π (0,1) i,k to promote sparse outcomes to be sampled with large probability.

3.3. PARAMETER ESTIMATION:

Parameter estimation of neural network models is typically done via maximum-likelihood estimation (MLE), where we approach a local minimum of the negative log-likelihood function via stochastic gradient descent with gradient computation automated by the back-propagation algorithm. Using the following shorthand notation:

EQUATION

)

The log-likelihood for a single data point can be formulated as:

log p(y|x, θ) = log l i=1 α(z i )β(f i )γ(y i )dzdf (8)

the computation of which is intractable. Instead, we resort to variational inference (VI; Jordan et al., 1999) , where we optimize a lower-bound on the log-likelihood

EQUATION

expressed with respect to an independently parameterized posterior approximation q(z, f |x, y, λ).

For as long as sampling from the posterior is tractable and can be performed via a reparameterization, we can rely on stochastic gradient-based optimization. In order to have a compact parameterization, we choose

EQUATION

)

This simplifies the lowerbound, which then takes the form of l nested expectations, the ith of which is E α(zi)β(fi) [log γ(y i )]. This is similar to the stochastic decoder of Schulz et al. (2018) , though our approximate posterior is in fact, also our parameterized prior. 6 Although this objective does not particularly promote sparsity, we employ sparsity-inducing regularization techniques that will be discussed in the next section.

Concretely, for a given source sentence x, target prefix y <i , and a latent sample z ≤i , f ≤i , we obtain a single-sample estimate of the loss by computing L i (θ) = -log γ(y i ).

3.4. REGULARIZATION:

In order to promote sparse distributions for the inflectional features, we apply a regularizer inspired by expected L 0 regularization (Louizos et al., 2018) . Whereas L 0 is a penalty based on the number of non-zero outcomes, we design a penalty based on the expected number of continuous outcomes, which corresponds to π (0,1) i,k as shown in Equation ( 6). For a given source sentence x, target prefix y <i , and a latent sample z <i , f <i , we aggregate this penalty for each feature

EQUATION

and add it to the cost function with a positive weight ρ. The final loss of the NMT model is

L(θ|D) = x,y∼D |y| i=1 L i (θ) + ρR i (θ) .

(12)

3.5. PREDICTIONS:

In our model, obtaining the conditional likelihood for predicting the most likely hypothesis requires marginalisation of the latent variables, which is intractable. An alternative approach is to heuristically search through the joint distribution,

EQUATION

rather than the marginal, an approximation that has been referred to as Viterbi decoding (Smith, 2011) . During beam search, we populate the beam with alternative target words, and for each prefix y <i in the beam, we resort to deterministically choosing the latent variables based on a single sample which we deem representative of their distributions, which is a common heuristic in VAEs for translation (Zhang et al., 2016; Schulz et al., 2018) . For unimodal distributions, such as the Gaussian p(z i |x, y <i , z <i , f <i ), we use the analytic mean, whereas for multimodal distributions, such as the Hard Kumaraswamy p(f i |x, y <i , z ≤i , f <i ), we use the argmax. 7

4. EVALUATION:



4.1. MODELS:

We evaluate our model by comparing it in machine translation against three baselines which constitute the conventional open-vocabulary NMT methods, including architectures using atomic parameterization either with subword units segmented with BPE (Sennrich et al., 2016) or characters, and the hierarchical parameterization method employed for generating all words in the output. We implement all architectures using Pytorch (Paszke et al., 2017) within the OpenNMT-py framework (Klein et al., 2017) 8 .

4.2. DATA AND LANGUAGES:

In order to evaluate our model we design two sets of experiments. The experiments in §4.4.1 aim to evaluate different methods under low-resource settings, for languages with different morphological typology. We model the machine translation task from English into three languages with distinct morphological characteristics: Arabic (templatic), Czech (fusional), and Turkish (agglutinative). We use the TED Talks corpora (Cettolo, 2012) for training the NMT models for these experiments. In §4.4.3, we conduct more experiments in Turkish to demonstrate the case of increased data sparsity using multi-domain training corpora, where we extend the training set using corpora from EU Bookshop (Skadin ¸š et al., 2014) , Global Voices, Gnome, Tatoeba, Ubuntu (Tiedemann, 2012) , KDE4 (Tiedemann, 2009) , Open Subtitles (Lison & Tiedemann, 2016) and SETIMES (Tyers & Alperen, 2010) 9 . The statistical characteristics of the training sets are given in Tables 4 and 5 . We use the official evaluation sets of the IWSLT 10 for validating and testing the accuracy of the models. In order to increase the number of unknown and rare words in the evaluation sets we measure accuracy on large test sets combining evaluation sets from many years (Table 6 presents the evaluation sets used for development and testing). The accuracy of each model output is measured using BLEU (Papineni et al., 2002) and chrF3 (Popović, 2015) metrics, whereas the significance of the improvements are computed using bootstrap hypothesis testing (Clark et al., 2011) . In order to measure the accuracy in predicting the correct syntactic description of the references, we also compute BLEU 7 We maximize across the three configurations of each feature, namely, max{π

{0} i,k , π {1} i,k , π (0,1) i,k }. If π (0,1) i,k

is highest, we return the mean of the underlying Kumaraswamy variable. 8 Our software is available at: https://github.com/d-ataman/lmm 9 The size of the resulting combined corpora is further reduced to filter out noise and reduce the computational cost of the experiments using data selection methods (Cuong & Simaan, 2014) . 10 The International Workshop on Spoken Language Translation scores over the output sentences segmented using a morphological analyzer. We use the AlKhalil Morphosys (Boudchiche et al., 2017) for segmenting Arabic, Morphidata (Straková et al., 2014) for segmenting Czech and the morphological lexicon model of Oflazer (Oflazer & Kuruöz, 1994) and disambiguation tool of Sak (Sak et al., 2007) for segmenting Turkish sentences into sequences of lemmas and morphological features.

4.3. TRAINING SETTINGS:

All models are implemented using gated recurrent units (GRU) (Cho et al., 2014) , and have a singlelayer bi-RNN encoder. The source sides of the data used for training all NMT models, and the target sides of the data used in training the subword-level NMT models are segmented using BPE with 16,000 merge rules. We implement all decoders using a comparable number of GRU parameters, including 3-layer stacked-GRU subword and character-level decoders, where the attention is computed after the 1 st layer (Barone et al., 2017) and a 3-layer hierarchical decoder which implements the attention mechanism after the 2 nd layer. All models use an embedding dimension and GRU size of 512. LMM uses the same hierarchical GRU architecture, where the middle layer is augmented using 4 multi-layer perceptrons with 256 hidden units. We use a lemma vector dimension of 150, 10 inflectional features (See §A.3 for experiments conducted to tune the feature dimensions) and set the regularization constant to ρ = 0.4. All models are trained using the Adam optimizer (Kinga & Ba, 2014) with a batch size of 100, dropout rate of 0.2, learning rate of 0.0004 and learning rate decay of 0.8, applied when the perplexity does not decrease at a given epoch. 11 Translations are generated with beam search with a beam size of 5, where the hierarchical models implement the hierarchical beam search algorithm (Ataman et al., 2019).

4.4. RESULTS:



4.4.1. THE EFFECT OF MORPHOLOGICAL TYPOLOGY:

The experiment results given in Table 1 show the performance of each model in translating English into Arabic, Czech and Turkish. In Turkish, the most sparse target language in our benchmark with rich agglutinative morphology, using character-based decoding shows to be more advantageous compared to the subword-level and hierarchical models, suggesting that increased granularity in the vocabulary units might aid in better learning accurate representations under conditions of high data sparsity. In Arabic, on the other hand, using a hierarchical decoding model shows to be advantageous compared to the subword and character-level models, as it might be useful in better learning syntactic dependencies. LMM obtains improvements of 0.51 and 0.30 BLEU points in Arabic and Turkish over the best performing baselines, respectively. The fact that our model can efficiently work in both Arabic and Turkish confirms that it can handle the generation of both concatenative and non-concatenative morphological transformations. The results in the English-to-Czech translation direction do not indicate a specific advantage of using either method for generating fusional morphology, where morphemes are already optimized at the surface level, although our model is still able to achieve translation accuracy comparable to the character and subword-level models.

4.4.2. PREDICTING UNSEEN WORDS:

In addition to the general machine translation evaluation using automatic metrics, we perform a more focused statistical analysis to illustrate the performance of different methods in predicting unseen words by computing the average perplexity per character on the input sentences which contain outof-vocabulary (OOV) words as suggested by Cotterell et al. (2018) . We also analyze the outputs generated by each decoder in terms of the frequency of unknown words in each model output and the Kullback-Leibler (KL) divergence between the character trigram distributions of the references and outputs, which represents the coherence between the statistical distribution learned by each model and the reference translations.

Our analysis results generally confirm the advantage of increased granularity during the generation of unseen words, where the character-level decoder can generate a higher rate of unseen word forms and higher KL-divergence with the reference, suggesting superior ability in generalizing to new Due to its stochastic nature, our model yields higher perplexity values compared to the hierarchical model, whereas the values range between subword and character-based models, possibly finding an optimal level of granularity between the two solutions. The KL-divergence and OOV rates confirm that our model has the potential in better generalize to new word forms as well as different morphological typology. Table 2 : Percentage of out-of-vocabulary (OOV) words in the output, normalized perplexity measures (PPl) per characters and the KL divergence between the reference and outputs of systems trained with in-domain data on different language directions.

4.4.3. THE EFFECT OF DATA SIZE:

Repeating the experiments in the English-to-Turkish translation direction by increasing the amount of training data with multi-domain corpora demonstrates a more challenging case, where there is a greater possibility of observing new words in varying context, either in the form of morphological inflections due to previously unobserved syntactic conditions, or a larger vocabulary extended with terminology from different domains. In this experiment, the character-level model experiences a drop in performance and its accuracy is much lower than the subword-level one, suggesting that its capacity cannot cope with the increased amount of training data. Empirical results suggest that with increased capacity, character-level models carry the potential to reach comparable performance to subword-level models (Cherry et al., 2018) . On the other hand, our model reaches a much larger improvement of 0.82 BLEU points over the subword-level and 2.54 BLEU points over the character-level decoders, suggesting that it could make use of the increased amount of observations for improving the translation performance, which possibly aid the morphology model in becoming more accurate.

4.4.4. THE IMPACT OF INFLECTIONAL FEATURES:

In order to understand whether the latent inflectional features in fact capture information about variations related to morphological transformations, we first try generating different surface forms of the same lemma by sampling a lemma vector with LMM for the input word 'go' and generating outputs using the fixed lemma vector and assigning different values to the inflectional features. In the second experiment, we assess the impact of the inflectional features by setting all features f to 0 and translating a set of English sentences with varying inflected forms in Turkish. Table 3 presents different sets of feature values and the corresponding outputs generated by the decoder and the outputs generated with or without the inflectional component.

Features Output English Translation [1,1,1,1,1,1,1,1,1,1] git go (informal) [0,1,1,1,1,1,1,1,1,1] 'a git to go [0,1,0,1,1,1,1,1,1,1]

'da git at go [0,0,0,1,1,0,0,1,1,0] gidin go (formal) [1,1,0,0,0,0,1,0,1,1] gitmek to go (infinitive) [0,0,1,0,0,0,0,0,0,1] gidiyor (he/she/it is) going [0,0,0,0,0,0,0,0,1,0] gidip by going (gerund) [0,0,1,1,0,0,1,0,1,0] gidiyoruz (we are) going

Input:

Output with f Output without f he went home. eve gitti. eve gitti. he came from home. evden geldi. eve geldi. it is good to be home. evde olmak iyi. evde olmak iyi. his home has red walls. evinde kırmızı duvarlar var. evde kırmızı duvar var. The model generates different surface forms for different sets of features, confirming that the latent variables represent morphological features related to the infinitive form of the verb, as well as its formality conditions, prepositions, person, number and tense. Decoding the set of sentences given in the second experiment LMM always generates the correct inflectional form, although when the feature values are set to 0 the model omits some inflectional features in the output, suggesting that despite partially relying on the source-side context, it still encodes important information for generating correct surface forms in the inflectional features.

5. CONCLUSION:

In this paper we presented a novel decoding architecture for NMT employing a hierarchical latent variable model to promote sparsity in lexical representations, which demonstrated promising application for morphologically-rich and low-resource languages. Our model generates words one character at a time by composing two latent features representing their lemmas and inflectional features. We evaluate our model against conventional open-vocabulary NMT solutions such as subword and character-level decoding methods in translationg English into three morphologically-rich languages with different morphological typologies under low to mid-resource settings. Our results show that our model can significantly outperform subword-level NMT models, whereas demonstrates better capacity than character-level models in coping with increased amounts of data sparsity. We also conduct ablation studies on the impact of feature variations to the predictions, which prove that despite being completely unsupervised, our model can in fact manage to learn morphosyntactic information and make use of it to generalize to different surface forms of words. The middle row shows the result of stretching it to include 0 and 1 in its support. The bottom row shows the result of rectification: probability mass under (l, 0) collapses to 0 and probability mass under (1, r) collapses to 1, which cause sparse outcomes to have non-zero mass. Varying the shape parameters (a, b) of the underlying continuous distribution changes how much mass concentrates outside the support (0, 1) in the stretched density, and hence the probability of sampling sparse outcomes.

A APPENDIX:



:

Notation We use capital Roman letters for random variables (and lowercase letters for assignments). Boldface Roman letters are reserved for neural network output vectors, and stands for elementwise multiplication. Finally, we denote typical neural network layers as layer(inputs; parameters).

In practice, we sample zi via a reparameterization in terms of a fixed Gaussian, namely, zi = ui + i si for i ∼ N (0, I d ). This is known as the reparameterization trick (Kingma & Welling, 2013), which allows back-propagation through stochastic units(Rezende et al., 2014).

Formally, because the decoder is an RNN, we are also conditioning on z<i and f<i. We omit this dependence to avoid clutter.

We use l = -0.1 and r = 1.1. Figure2in the appendix illustrates different instances of this distribution.

This means we refrain from conditioning on the observation yi itself when sampling zi and fi. Whereas this gives our posterior approximation access to less features than the true posterior would have, we do not employ a fixed uninformative prior, but rather an autoregressive network trained on the likelihood of generated word forms. This is a common approximation for latent variable models that employ autoregressive priors(Goyal et al., 2017).

Perplexity is the exponentiated average negative log-likelihood per segment (BPE, or character) that a model assigns to a dataset. It corresponds to the model's average surprisal per time step.

Figures and Tables:

Figure 2: The top row shows the density function of the continuous base distribution over (0, 1).The middle row shows the result of stretching it to include 0 and 1 in its support. The bottom row shows the result of rectification: probability mass under (l, 0) collapses to 0 and probability mass under (1, r) collapses to 1, which cause sparse outcomes to have non-zero mass. Varying the shape parameters (a, b) of the underlying continuous distribution changes how much mass concentrates outside the support (0, 1) in the stretched density, and hence the probability of sampling sparse outcomes.

Above: Machine translation accuracy in Arabic (AR), Czech (CS) and Turkish (TR) in terms of BLEU and ChrF3 metrics as well as BLEU scores computed on the output sentences tagged with the morphological analyzer (t-BLEU) using in-domain training data. Below: The performance of models trained with multi-domain data. Best scores are in bold. All improvements over the baselines are statistically significant (p-value < 0.05).output and not necessarily copying previous observations as the subword-level model, however, this advantage is more visible in Turkish and less in Czech or Arabic. The hierarchical decoder which performs the search at the level of words, on the other hand, behaves with less uncertainty in terms of the perplexity values although it cannot demonstrate the ability to generalize to new forms and neither can closely capture the actual distribution in the target language.

Table content: <table><tr><td>(only in-domain)</td></tr></table>

Above: Outputs of LMM based on the lemma 'git' ('go') and different sets of inflectional features. Below: Examples of predicting inflections in context with or without using features.

Table content: <table/>

A.1 THE STATISTICAL CHARACTERISTICS OF EXPERIMENTAL DATA Training sets based on the TED Talks corpora (M: Million, K: Thousand).

Table content: <table><tr><td colspan="2">Language Pair # sentences</td><td colspan="2"># tokens</td><td colspan="2"># types</td></tr><tr><td/><td/><td colspan="4">Source Target Source Target</td></tr><tr><td>English-Arabic</td><td>238K</td><td>5M</td><td>4M</td><td>120K</td><td>220K</td></tr><tr><td>English-Czech</td><td>118K</td><td>2M</td><td>2M</td><td>50K</td><td>118K</td></tr><tr><td>English-Turkish</td><td>136K</td><td>2M</td><td>3M</td><td>53K</td><td>171K</td></tr><tr><td colspan="2">Language Pair # sentences</td><td colspan="2"># tokens</td><td colspan="2"># types</td></tr><tr><td/><td/><td colspan="4">Source Target Source Target</td></tr><tr><td>English-Turkish</td><td>434K</td><td>8M</td><td>6M</td><td>135K</td><td>373K</td></tr></table>

The multi-domain training set (M: Million, K: Thousand).

Table content: <table><tr><td>Language</td><td>Data sets</td><td># sentences</td></tr><tr><td colspan="2">English-Arabic Development dev2010, test2010</td><td>6K</td></tr><tr><td/><td>test2011, test2012</td><td/></tr><tr><td>Testing</td><td>test2013, test2014</td><td>4K</td></tr><tr><td colspan="2">English-Czech Development dev2010, test2010,</td><td>3K</td></tr><tr><td/><td>test2011</td><td/></tr><tr><td>Testing</td><td>test2012, test2013</td><td>3K</td></tr><tr><td colspan="2">English-Turkish Development dev2010, test2010</td><td>3K</td></tr><tr><td>Testing</td><td>test2011, test2012</td><td>3K</td></tr></table>

Development and testing sets (K: Thousand).

Table content: <table><tr><td>A.2 THE KUMARASWAMY DISTRIBUTION</td></tr></table>

Peer review (iclr2020):
This paper addresses the problem of translating into morphologically-rich languages, which suffers from the problem of sparse vocabularies and high numbers of rare and unseen words. In particular, current approaches such as subwords lack explicit notions of morphology and are obtained independently of the translation objective, while operating at the character-level renders the learning of long-distance dependencies more difficult. More concretely, this paper models the generation of target words in a stochastic, hierarchical process, where the morphological features are modelled as latent variables. At each target word, the model samples a vector representing its lemma, followed by a k-dimensional latent inflectional features (e.g. nominative or accusative). To induce sparsity in the inflectional features, the paper uses a *stretch-and-rectify* distribution (Louizos et al., 2018) using the Kumaraswamy distribution. The paper further applies a sparsity-inducing regulariser to encourage the inflectional features to take discrete values of *0* or *1*. Parameter estimation is done by optimising a lower-bound on the marginal log-likelihood. Experiments on translations into three morphologically-rich languages, English-{Arabic, Czech, Turkish}, indicate fairly small but consistent and statistically significant gains in BLEU. Further analysis indicates improved perplexity upper bound on rare words compared to subword-based baselines, along with somewhat interpretable latent inflectional features. Overall, this paper proposes an elegant solution to an important problem, and yields statistically significant BLEU improvements over the baselines. I have listed some pros and cons below, with a recommendation of *Weak Accept*. I would be willing to raise my scores assuming my concerns are sufficiently addressed. Pros: 1. The proposed approach is easy-to-follow and explained clearly, and the paper is overall well-written. 2. Experiments are done on translation into three morphologically-rich languages with both concatenative and non-concatenative morphology, and results in small but mostly consistent BLEU improvements over the best subword, hierarchical, and character baselines for each language. 3. The proposed approach is general and can potentially be applied on top of other model architectures (e.g. Transformers) and other language generation problems, such as language modelling, summarisation, etc, although the experiments in the paper focus on NMT with a GRU-based architecture. Cons: 1. It is still unclear how much computational overhead is introduced by the approach (in terms of both training and prediction times), and how scalable the approach is when applied to language pairs that have much bigger training sets. In Tables 4 and 5, it seems that the largest dataset is English-Turkish multi-domain (434K sentence pairs), which is fairly small compared to other datasets. 2. Regarding the feature variations result in section 4.4.4, it is hard to draw any conclusions about the latent inflectional features just based on varying the inflectional features at one position. I would suggest running the same experiment for multiple words at different positions, and see whether the same set of inflection features always results in similar inflections. This would better convince the reader that the latent inflectional features really are capturing useful morphological information, and that the target word generation process is appropriately controlled by the latent variable. 3. Since some of the experiments did not use the same standard splits (e.g. the multi-domain Turkish experiments), it would be nice to report how well external models (e.g. OpenNMT) would do in the authors* dataset split, to make sure that the reported numbers here are at least comparable to that. This would help ensure the credibility of the findings. 4. The effect of data size experiments (Section 4.4.2) for the character model seems somewhat counter-intuitive. Why would making the training set bigger (by incorporating data from a more diverse domain) make the character model worse? The explanation that *[the character model]*s capacity cannot cope with the increased amount of sparsity* does not seem satisfactory. Why does adding more data result in increased sparsity? Furthermore, one can simply use more hidden units or deeper layers to mitigate this problem in the character-based model. Assuming the authors are controlling for model capacity, then the character model*s hidden state can be made bigger (since the vocabulary size is lower for character models, thus the embedding layer and the softmax layer are by definition smaller than the subword-based model). Minor suggestions: 1. Section 3.5 is not very clear; adding some figures or more explanation there would help understand how prediction is done. 2. It would be interesting to explore other potential metric for measuring morphological generalisation. For instance, at evaluation time, does the model predict words that it has never seen before (e.g. predicting *plays*, even though the model has only seen *play* at training time) at a higher frequency than character/subword/hierarchical baselines? If yes, this would provide more evidence that the latent inflectional features are properly capturing morphological information.