A Latent Morphology Model for Open-Vocabulary Neural Machine Translation
Authors: Duygu Ataman
Wilker Aziz
Alexandra Birch
Venue:  (2020)
Keywords: neural machine translation
low-resource languages
latent-variable models

Abstract: Translation into morphologically-rich languages challenges neural machine translation (NMT) models with extremely sparse vocabularies where atomic treatment of surface forms is unrealistic. This problem is typically addressed by either pre-processing words into subword units or performing translation directly at the level of characters. The former is based on word segmentation algorithms optimized using corpus-level statistics with no regard to the translation task. The latter learns directly from translation data but requires rather deep architectures. In this paper, we propose to translate words by modeling word formation through a hierarchical latent variable model which mimics the process of morphological inflection. Our model generates words one character at a time by composing two latent representations: a continuous one, aimed at capturing the lexical semantics, and a set of (approximately) discrete features, aimed at capturing the morphosyntactic function, which are shared among different surface forms. Our model achieves better accuracy in translation into three morphologically-rich languages than conventional open-vocabulary NMT methods, while also demonstrating a better generalization capacity under low to mid-resource settings.

Published as a conference paper at ICLR 2020
A LATENT MORPHOLOGY MODEL FOR OPEN- VOCABULARY NEURAL MACHINE TRANSLATION
Duygu Ataman" University of Zürich ettaan@cl.ure .uzh..h... ch
Wilker Aziz University of Amsterdam a z zz z@uva. m1
Alexandra Birch
University of Edinburgh a b rcchedd.. ac. uk
ABSTRACT
Translation into morphologicall-rrich languages challenges neural machine translation (NMT) models with extremely sparse vocabularies where atomic treatment of surface forms is unrealistic. This problem is typically addressed by either pre-processing words into subword units or performing translation directly at the level of characters. The former is based on word segmentation algorithms optimized using corpus-level statistics with no regard to the translation task. The latter learns directly from translation data but requires rather deep architectures. this paper, we propose to translate words by modeling word formation through a hierarchical latent variable model which mimics the process of morphological inflection. Our model generates words one character at a time by composing two latent representations: a continuous one, aimed capturing the lexical semanties, and a set of (approximately) discrete features, aimed at capturing the morphosyn- tactic function, which are shared among different surface forms. Our model languages
achieves better accuracy in translation into three morphologically than conventional open-vocabulary NMT methods, while also demonstrating a better generalization capacity under low to mid-resource settings.
INTRODUCTION
Neural machine translation (NMT) models are conventionally trained by maximizing the likelihood of generating the target side a bilingual parallel corpus of observations one word at a time condi- tioned of their full observed context. NMT models must therefore learn distributed representations that accurately predict word forms in very diverse contexts, a process that is highly demanding in terms of training data well as the network capacity. Under conditions of lexical sparsity, which includes both the case of unknown words and the case of known words occurring in surprising con- texts, the model is likely to struggle. Such adverse conditions are typical of translation involving morphologically-ricc languages, where any single root may lead to exponentially many different surface realizations depending on its syntactic context. Such highly productive processes of word formation lead to many word forms being rarely or ever observed with a particular set of mor- phosyntactic attributes. The standard approach overcome this limitation is pre-process words into subword units that are shared among words, which are, in principle, more reliable as they are observed more frequently in varying context (Sennrich et al., 2016 Wu et al.) (016)). One draw- back related to this approach, however, is that the estimation of the subword vocabulary relics on word segmentation methods optimized using corpus-dependent statistics, disregarding any linguistic notion of morphology and the translation objective. This often produces subword units that are mantically ambiguous as they might be used in far too many lexical and syntactic contexts (Ataman) al.) 2017). Moreover, in this approach, word form is then generated by prediction of multiple subword units, which makes generalizing to unseen word forms more difficult due to the possibility that a subword unit necessary to reconstruct a given word form may be unlikely in given context. To alleviate the sub-optimal effects of using explicit segmentation and generalize better to new mor- phological forms, recent studies explored the idea of extending NMT model translation directly at
Work done while the first author was doctoral student at the University of Trento and visiting post- graduate student the University of Edinhurgh.

---

Published as a conference paper at ICLR 2020
the level of characters (Kreutzer & Sokolov 2018 Cherry et al. 2018), which, in turn, have demon- strated the requirement of using comparably deeper networks, as the network would then need to learn longer distance grammatical dependencies (Sennrich, 2017).
In this paper, we explore the benefits of explicitly modeling variation in surface forms of words using techniques from deep latent variable modeling in order improve translation accuracy for low-resource and morphologically-rich languages. Latent variable models allow us to inject induc- tive biases relevant the task, which, in our case, is word formation during translation. In order to formulate the process of morphological inflection, design a hierarchical latent variable model which translates words one character at a time based on word representations learned composition- ally from sub-lexical components. In particular, for each word, our model generates two latent representations: i) a continuous- space dense vector aimed at capturing the lexical semanties of the word in given context, and ii) a set of (approximately) discrete features aimed at capturing that word's morphosyntatic role the sentence. We then see inflection as decoding word form, one character at a time, from learned composition of these two representations, By forcing the model to encode each word representation in terms of a more compact set of latent features, we encour- age them to be shared across contexts and word forms, thus, facilitating generalization under sparse settings. We evaluate our method in translating English into three morphologically-tie each with distinct morphological typology: Arabic, Czech and Turkish, and show that our model is able to obtain better translation accuracy and generalization capacity than conventional approaches to open-vocabulary NMT.
languages
NEURAL MACHINE TRANSLATION
In this paper, we use recurrent NMT architectures based on the model developed by Bahdanau et al. (014). The model essentially estimates the conditional probability of translating a source sequence (01, 22, ... Fm) into target sequence (y1 y2.... y1) via an exact factorization:
plyl., 8) II ply, (0,,,,,,e..00
where Yoi stands for the sequence preceding the ith target word. At each step of the sequence, a fixed neural network architecture maps its inputs, the source sentence and the target prefix, to the probability of the ith target word observation context. In order to condition on the source sentence fully, this network employs an embedding layer and bi-directional recurrent neural network (bi-RNN) based encoder. Conditioning on the target prefix V<s implemented using a recurrent neural network (RNN) based decoder, and an attention mechanism which summarises the source sentence into a context vector c; as a function of a given prefix (Luong et al., 2015). Given a parallel training set D, the parameters 0 of the network are estimated to attain a local minimum of the negative log- ligliiili function L(A[D) = gradient-based optimization (Bottou & Cun 2004).
- Ex log p(y x. f) via stochastic
Atomic parameterization estimates the probability of generating each target word y in a single shot:
exp h.) E exp hE
P(Uil,, Vei, 0)
in
(2)
where E E Rexd is the target embedding matrix and the decoder output h; E Rd represents and Vei. Clearly, the size e of the target vocabulary plays an important role in determining the complexity of the model, which creates important bottleneck when translating into low-resource and morphologically- languages due to the sparsity in the lexical distribution.
Recent studies approached this problem by performing NMT with subword units, popular one of which is based on the Byte-Pair Encoding algorithm (BPE; Sennrich et al., 2016), which finds the optimal description of corpus vocabulary by iteratively merging the most frequent character quences. Atomic parameterization could also be used to model translation at the level characters, which is found be advantageous in generalizing morphological variations (Cherry al. 2018).
2

---

Published as a conference paper at ICLR 2020
Hierarchical paramaterization further factorizes the probability of target word in context:
4.
V.ei, Meci 0) = II ply,,la, Meci Vi,<j: ==1
(3)
where the ith word y/ = (1 Vt.l,) is seen as a sequence of 4 characters. Generation follows one character a time, each with probability computed by fixed neural network architecture with varying inputs, namely, the source sentence the target prefix y<i, and the prefix yı,<j of characters already generated for that word. this case there are two recurrent cells, one updated at the boundary of each token, much like in the standard case, and another updated at the character level. Luong Manning (2016) propose hierarchical parameterization to compute the probability p(y. 2. Ve.. 0) for unknown words, while for known words they use the atomic parameterization. In this paper, we use the hierarchical parameterization method for generating all target words, where also augment the input embedding layer with character-level bi-RNN, which computes each word representation y; as a composition the embeddings of their characters (Ling et al. 2015).
A LATENT MORPHOLOGY MODEL (LMM) FOR LEARNING WORD REPRESENTATIONS
The application of a hierarchical structure for learning word representations in language modeling [Vania & Lopez 2017) or semantic role labeling (Sahin & Steedman 2018 have shown that such featuree representations encode many cues about the morphological features of words by establishing a map- ping between phonetic units and lexical context. Although it can provide alternative solution to open-vocabulary NMT by potentially alleviating the need for subword segmentation, the quality of word representations learned by an hierarchical model is still highly dependant on the amount of observations (Sahin & Steedman 2018 Ataman et al. (019). since the training data is essential in properly modeling the lexical distribution. On the other hand, the process of word formation, particularly morphological inflection, has many properties that remain universal across languages, where a word typically composed of lemma, representing its lexical semanties, and a distinct combination of categorical inflectional features expressing the word's syntactic role in the phrase or sentence. In this paper, we propose to manipulate this universal structure in order to enforce an inductive bias on the prior distribution of words and allow the hicrarchical parameterization model in properly learning lexical representations under conditions of data sparsity.
3.1 GENERATIVE MODEL
Our generative LMM for NMT formulates word formation in terms of stochastic process, where each word is generated one character at time by composing two latent representations: a continuous vector aimed at capturing the lemma, and set of sparse features aimed capturing the inflectional features. The motivation for using a stochastic model is twofold. First, deterministic models are by definition unimodal: when presented with the same input (the same context) they always produce the same output. When we model the word formation process, it is reasonable to expect a larger degree of ambiguity, that is, for the same context (e.g. a noun prefix), we may continue by inflecting the word differently depending on the (latent) mode of operation we are at (e.g. generating nomina- tive, accusative or dative noun). Second, stochastie models, the choice of distribution gives us a mechanism to favour a particular type of representation. In our case, we use sparse distributions for inflectional features to accommodate the fact. that morphosyntactic features are discrete in nature. Our latent variable model is an instance of variational auto-encoder (VAE; Kingma & Welling 2013 inspired by the model of Zhou & Neubig (201)) for morphological reinflection.
Generation of the ith word starts by sampling Gaussian-distributee representation in context. This requires predicting the Gaussian location u, and scale s; vectors !
Z |z, M<r (N,,, diag(s, @ s:))
u, dense(h; 0.)
S; Ç(dense 02))
Notation We use capital Roman letters for random variables (and lowercase letters for assignments). Boldface Roman letters are reserved for neural network output vectors, and @ stands for elementwise multipli- cation. Finally, denote typical neural network layers as (inputssn parameters).
3

---

Published as a conference paper at ICLR 2020
embeddings
encoderr
infeectional
W-ll
features
Character- level decoder
dense
attention
Character bi-RNN
next
lemma
G-1
Figure 1: LMM for computing word representations while translating the sentence '.. went home' into Turkish ('eve(tt))home girrilhe/she/ttmeent)) The character-level decoder initialized with the attentional vector h; computed by the attention mechanism using current context C₁ and the word representation as in Luong & Manning (2016).
where prediction of the location (in Rd) and scale (in RR from the word-level decoder hidden state h; (which represents x and Vees) is performed by two dense layers, and the scale values are ensured be positive with the softplus (() activation
Generation proceeds by then sampling a K K-dimensional vector fi of sparse scalar features (see $3.2) conditioned on the source x, the target prefix Ycii and the sampled lemma z₁. We model sampling of fi conditioned on 2; in order to capture the insight that inflectional transformations typically depend on the category of lemma. Having sampled fi and 24, the representation of the ith target word computed by transformation of 2; and fi, i.e. t, = dense([=;, fx]: cump)
As shown Figure I. our model generates each word character by character auto-regressively by conditioning on the word representation t; predicted by the LMM, the current context c₁. and the previously generated characters following the hierarchical parameterization See Algorithm for details on generation.
Input:
model parameters 8, latent lemma 2,, latent morphological attributes fú, observed character sequence "...,...,,,... if training or placeholders if test, decoder state h; and context vector C;
Result: updated decoder hidden state, prediction (a word), probability of prediction (for loss) initialization;
t, = dense([2s, fi]: foommp); initialize char-rnn with projection of [t;; c:]: for j < L and j < max do
compute output layer from char-rnn state
if training then
set prediction to observation Yi..
else
set prediction to arg max of output softmax layer; end
assess log-probability of prediction;
update word-level RNN decoder with prediction; end
Word generation: in training the word observed, thus we only update the de- coder and assess the probability of the observation, in test, mean values of the distribu- tions to represent most likely values for 2 and f and populate predictions with beam-search
Algorithm 1:
In practice, sample via reparameterization in of fixed Gaussian, namely, 2,
for é (0, Ia)- This known the reparameterization trick (Kingma & Welling 2013), which allows hack-propagation through stochastic units (Rezende et al. (0044). Forrmally because the decoder is an RNN, we also conditioning zei and fei. We omit this depen- dence avoid clutter.

---

Published as a conference paper at ICLR 2020
3.2 SPARSE FEATURES
Since each target word y; may have multiple inflectional features, ideally, we would like St to be K feature indicators, which could be achieved by sampling from K independent Bernoulli distribu- tions parameterized in context. The problem with this approach is that sampling Bernoulli outcomes non-differentiable, thus, their training requires gradient estimation via REINFORCE (Williams) 1992) and sophisticated variance reduction techniques. An alternative approach that has recently become popular is to use relaxations such as the Concrete distribution or Gunbel-Softmax (Maddi- son et al., 2017] Jang et al.] 2017) in combination with the straight-through estimator (ST; [Bengio ctt al. 2013), ). This is based on the idea of relaxing the discrete variable from taking on samples the discrete set {0, J to taking
samples in the continuous set (0, 1) using distribution for which reparameterization exists (e.g. Gumbel). Then, a -in-differentiable activation (e.g. a threshold func- tion) maps continuous outcomes to discrete ones. ST simply ignores the discontinuous activation in the backward pass, i.e. it assumes the Jacobian is the identity matrix. This does lead to biased estimates of the gradient of the loss, which is in conflict with the requirements behind stochastic optimization (Robbins & Monro 1951).
An alternative presented by Louizos et al. 2018) achieves different compromise, it gets rid of bias at the cost of mixing both sparse and dense outcomes. The idea is to obtain a continuous sample (0, 1) from a distribution for which a reparameterization exists and stretch it continuous support r)) 3 (0, 1) using a simple linear transformation = I + (r l)c. A rectifier is then employed map the negative outcomes to 0 and the positive outcomes larger than one to 1, i.e. = min(1, max(0, The rectifier is only non-differentiable at 8 0 and at = 1, however, because the stretched variable s is sampled from a continuous distribution, the chance of sampling O and 8 = 1 is essentially O. This seetched-and-reetiffee distribution allows: i) the sampling procedure to become differentiable with respect to the parameters of the distribution, ii) to sample sparse outcomes with unbiased estimator, and iii) to calculate the probability of sampling 0 and f = in closed form as a function of the parameters of the underlying distribution, which corresponds to the probability of sampling
and > 1, respectively.
In their paper, Louizos Louiz,, et al. (2018) used the BinaryConcrett (or Gumbel Sigmoodd) as the underlying continuous distribution, the sparsity of which is controlled via temperature parameter. However, in our study, we found this parameter difficult to predict, since it is very hard to allow a neural network to control its value without unstable gradient updates. Instead, we opt for slight variant by Bastings et al. (2019) based on the Kumaraswamy distribution (Kumaraswamy 1980), a two- parameters distribution that closely resembles Beta distribution and is sparse whenever its (strictly positive) parameters are between 0 and 1. In the context of text classification, Bastings et al. (2019) shows this stretch-and anch--nnd-eeeee recttcff technique to work better than methods based on REINFORCE.
For cach token ys, we sample K independent Kumaraswamy variables in context,
C₁ xr Dei, 2; ~ Kuma (,,,,, by.s) [a, b,] = [ [(dense[[z,, h:]: 0ms))
I....,K
which makes a continuous random vector e in the support (0, 1)K D We then stretch-and-reetify the samples via fi, min(1, max(0, / (r ()c,,)) making f. random vector in the support [0, 1] The probability that fix is exactly is
10)
Kuma(clas, d..k )d
(6a)
i
and the probability that fi, is exactly 1 is
(1)
Kuma Kuma(clo;,s, dy,k de
(6b)
J
and therefore the complement
π i,k
(0.1)
{0)
[1) 5
(6c)
'In practice we sample G, via a reparameterization of fixed uniform variable, namely, C1 = (1 (1 - s..)) (h,,,) 'A,,, 1/es, where εs,6 ~ 2/(0, 1), which much like the Gaussian reparameterization enables back- propagation through samples (Nalisnick & Smyth 2016).
We use I = -0.1 and T.1. Figure in the appendix illustrates different instances of this distribution.
5

---

Published as a conference paper at ICLR 2020
the probability that fi,& be any continuous value in the open set (0, 1). In $3.4 will derive
regularizers based on i,k
(0.1) to promote sparse outcomes to be sampled with large probability.
3.3 PARAMETER ESTIMATION
Parameter estimation neural network models is typically done via maximum-likeliinee estimation (MLE), where approach a local minimum of the negative log-likelihood function via stochastic gradient descent with gradient computation automated by the back-propagation algorithm. Using the following shorthand notation:
a(zi) _ p(zz lr, Mei Reis 0c
(7a)
K
B(G) _ II Wes Res k=1
(7b)
7()) A
II p(us,
Ia, Vea, 2gi, fgg Ui, 0)
(7c)
j=1
The log-likelihood for single data point can be formulated as:
log logp(y|z, 0) = log II i=1
[((x))(())(u)ddd))
(8)
the computation of which is intractable. Instead, we resort to variational inference (VI: Jordan et al. 1999). where we optimize lower-bound on the log- liklliill
log a(=))((f))((s))
{2,,f1zAA
expressed with respect an independently parameterized posterior approximation g(z, f|x, y, A). For as long as sampling from the posterior is tractable and can be performed via a reparameterization, we can rely on stochastic gradient-bai based optimization. order have a compact parameterization, we choose
q2, flz y, -I o(z))((1))
(10)
This simplifies the lowerbound, which then takes the form l nested expectations, the which is (a()))(()) (u:)). This is similar to the stochastic decoder of Schulz et al. (2018), though approximate posterior is in fact, also parameterized prior Although this objective does not particularly promote sparsity, we employ pparsity-inducing regularization techniques that will be discussed in the next section.
Concretely, for a given source sentence s, target prefix yeu. and a latent sample zgi, féi. we obtain single-sample estimate of the loss by computing (,) = - log (11).
3.4 REGULARIZATION
In order to promote sparse distributions for the inflectional features, we apply a regularizer inspired by expected To regularization (Louizos al. 2018). Whereas Lo is penalty based the number of non-zero outcomes, we design a penalty based on the expected number of continuous outcomes, (0.1)
which corresponds to πī.k shown Equation (6. For given source sentence æ, target prefix Vei. and latent sample z<i; fei aggregate this penalty for each feature
(+)
(0.1) T.kk
(11)
"This means refrain from conditioning on the observation y4 itself when sampling 2, and Whereas this gives our posterior approximation access to less features than the true posterior would have, we do not employ fixed uninformative prior, but rather an autoregressive network trained on the likelihood of generated word forms. This is common approximation for latent variable models that employ autoregressive priors (Goyall et 2017)

---

Published as a conference paper at ICLR 2020
and add it to the cost function with a positive weight p. The final loss of the NMT model is
l
[(0ID)
= Σ
Σ E.ya ==1
(12)
3.5 PREDICTIONS
In our model, obtaining the conditional likelihood for predicting the most likely hypothesis requires marginalisation of the latent variables, which is intractable. An alternative approach is heuristi- cally search through the joint distribution,
arg max y.s.f
(13)
rather than the marginal, an approximation that has been referred to as Viterbi decoding (Smith 2011]. During beam search, we populate the beam with alternative target words, and for each prefix yei in the beam, we resort to deterministically choosing the latent variables based on single sample which we deem representative of their distributions, which is common heuristic in VAEs for translation (Zhang et al. 2016; Schulz et al., 2018). For unimodal distributions, such as the Gaussian p(z; | [r, M-ci R-ci Je)) we use the analytic mean, whereas for multimodal distributions, such the Hard Kumaraswamy p(f [r, Vea: Zgas f<i), we use the argmax
4 EVALUATION
4.1 MODELS
We evaluate our model by comparing in machine translation against three baselines which con- stitute the conventional open-vocabulary NMT methods, including architectures using atomic pa- rameterization either with subword units segmented with BPE (Sennrich et al. (2016) or characters, and the hierarchical parameterization method employed for generating all words in the output. We implement all architectures using Pytorch (Paszke et al. 2017) within the OpenNMT-py framework KKlein al. 2017
4.2
DATA AND LANGUAGES
In order to evaluate model we design two sets of experiments. The experiments in $4.4.1 aim to evaluate different methods under low-resource settings, for languages with different morphological typology. We model the machine translation task from English into three languages with distinct morphological characteristics: Arabic (templatic), Czech (fusional), and Turkish (agglutinatiie). We use the TED Talks corpora (Cettolo 2012) for training the NMT models for these experiments. In $4.4.3. we conduct more experiments in Turkish to demonstrate the case of increased data sparsity using multi-domain training corpora, where we extend the training set using corpora from EU Book- shop (Skadins et al. 2014), Global Voices, Gnome, Tatoeba, Ubuntu (Tiedemann, 2012). KDE4 [Tedemann) 2009). J Open Subtitles (Lison & Tiedemann 2016) and SETIMES (Tyers & Alperen 2010 The statistical characteristics of the training sets are given in Tables 4 and 5 We use the official evaluation sets of the [WSL1 for validating and testing the accuracy of models. In order to increase the number of unknown and rare words in the evaluation sets we measure accuracy on large test sets combining evaluation sets from many years (Table 6 presents the evaluation sets used for development and testing). The accuracy of each model output is measured using BLEU (Papineni et al. 2002) and chrF3 (Popović) 2013) metrics, whereas the significance of the improve- ments are computed using bootstrap hypothesis testing (Clark et al., 2011). In order to measure the accuracy in predicting the correct syntactic description of the references, we also compute BLEU
(0.1) (0) (1) (0.1)) If m WWe maximize across the three configurations of each feature, namely, max T..k T highest, we return the mean of the underlying Kumaraswamy variable.
"Our software is available at: https:/ /ii gitt.bb. htt///daaaaaa///na
resulting combined corpora is further reduced to filter out noise and reduce the computa- tional cost of the experiments using data selection methods (Cuong & Simaan 2014).
"The size
The International Workshop Spoken Language Translation
7

---

Published as a conference paper at ICLR 2020
scores over the output sentences segmented using morphological analyzer. We use the AlKhalil Morphosys [Boudchiche et al. 2017) for segmenting Arabic, Morphidata (Straková et al. 2014) for segmenting Czech and the morphological lexicon model of Oflazer (Oilazer & Kuruöz 1994) and disambiguation tool of Sak (Sak et al., 2007) for segmenting Turkish sentences into sequences of lemmas and morphological features.
4.3 TRAINING SETTINGS
All models are implemented using gated recurrent units (GRU) (Cho et al. 2014), and have single- layer bi-RNN encoder. The source sides of the data used for training all NMT models, and the target sides of the data used in training the subword-level NMT models are segmented using BPE with 16,000 merge rules. We implement all decoders using a comparable number of GRU parameters, including 3-layer stacked-GRU subword and character-level decoders, where the attention is com- puted after the 1s layer (Barone et al. 2017) and a 3-layer hierarchical decoder which implements the attention mechanism after the 2nd layer. All models use an embedding dimension and GRU size of 512. LMM uses the same hierarchical GRU architecture, where the middle layer is augmented using 4 multi-layer perceptrons with 256 hidden units. We use lemma vector dimension of 150, 10 inflectional features (See |AA3 for experiments conducted to tune the feature dimensions) and set the regularization constant to p = 0.4. All models are trained using the Adam optimizer (Kinga & Bal 2014) with a batch size of 100, dropout rate of 0.2, learning rate of 0,0004 and learning rate decay of 0.8, applied when the perplexity does not decrease at a given epoch Translations are generated with beam search with beam size of 5, where the hierarchical models implement the hierarchical beam search algorithm (Ataman et al. 2019).
4.4
RESULTS
4.4.1 THE EFFECT OF MORPHOLOGICAL TYPOLOGY
The experiment results given in Table [ show the performance of each model in translating En- glish into Arabic, Czech and Turkish. In Turkish, the most sparse target language in our benchmark with rich agglutinative morphology, using character- decoding shows to be more advantageous compared to the subword-lewe and hierarchical models, suggesting that increased granularity in the vocabulary units might aid in better learning accurate representations under conditions of high data sparsity. In Arabie, on the other hand, using a hierarchical decoding model shows to be advanta- geous compared to the subword and character-level models, as it might be useful in better learning syntactic dependencies. LMM obtains improvements of 0.51 and 0.30 BLEU points in Arabic and Turkish over the best performing baselines, respectively. The fact that our model can efficiently work in both Arabic and Turkish confirms that it can handle the generation of both concatenative and non-concatenative morphological transformations. The results in the English-to-Czes lation direction do not indicate a specific advantage of using either method for generating fusional morphology, where morphemes are already optimized at the surface level, although our model is still able to achieve translation accuracy comparable to the character and subword-level models.
trans-
4.4.2 PREDICTING UNSEEN WORDS
In addition to the general machine translation evaluation using automatic metrics, we perform more focused statistical analysis to illustrate the performance of different methods in predicting unseen words by computing the average perplexity per character on the input sentences which contain out- of-vocabulary (OOV) words as suggested by Cotterell et al. (201)). We also analyze the outputs generated by each decoder in terms of the frequency of unknown words each model output and the Kullback-Leibler (KL) divergence between the character trigram distributions of the references and outputs, which represents the coherence between the statistical distribution learned by each model and the reference translations.
Our analysis results generally confirm the advantage of increased granularity during the generation of unseen words, where the character-level decoder can generate higher rate of unseen word forms and higher KL-divergence with the reference, suggesting superior ability in generalizing to new
Perplexity is the exponentiated average negative log-likelihood per segment (BPF, character) that model assigns dataset. It corresponds the model's average surprisal per time step.
8

---

Published as a conference paper at ICLR 2020
(only in (n-domiin)
Model
AR t-BLEU
CS
BLEU
chrF3
BLEU
t-BLEU
chrF3
BLEU
t-BLEU
chrF3
Subwords
14.27
51.24
0.3927
16.60
54.22
0,4123
8.52
38.03
0.3763
Chars
12.72
47.56
0.3804
16.94
52.80
0.4103
10.63
40.63
0.3810
Hierarch.
15.55
54.01
0.4154
16.79
48.27
0,4068
9.74
35,91
0.3771
LMM
16.06
55.97
0.4251
16.97
50.35
0,4095
10.93
45.47
0.3889
(multi-domain)
Model
TR
BLEU
t-BLEU
chrF3
Subwords
10.42
42.65
0.3722
Charss
8,94
37.12
0.3274
Hierarch. 10.35
40.54
0,3870
LMM
11.48
48.23
0.3939
Table I: Above: Machine translation accuracy in Arabic (AR), Czech (CS) and Turkish (TR) in terms of BLEU and ChrF3 metrics as well as BLEU scores computed on the output sentences tagged with the morphological analyzer (t-BLEU) using in-domain training data. Below: The performance of models trained with multi-domain data. Best scores are in bold. All improvements over the baselines are statistically significant (p-value < 0.05).
output and not necessarily copying previous observations as the subword-level model, however, this advantage is more visible in Turkish and less in Czech or Arabic. The hierarchical decoder which performs the search at the level of words, on the other hand, behaves with less uncertainty in terms of the perplexity values although it cannot demonstrate the ability to generalize to new forms and neither can closely capture the actual distribution the target language.
Due to its stochastic nature, our model yields higher perplexity values compared to the hierarchical model, whereas the values range between subword and bharacter-based models, possibly finding an optimal level of granularity between the two solutions. The KL-divergence and OOV rates con- firm that our model has the potential better generalize to new word forms as well as different morphological typology.
Model
AR
CS
TR
00V%
Ppl
KL-Div
00V%
Ppl
KL-Div
00V%
Ppl
KL-Div
Subwords
1.75
2.84
12,871
2.39
2.62
8,954
3.54
2.78
17,342
Char.s
3.08
2.46
29,607
1.90
2.61
17,092
4.28
2.38
38,043
Hierarch. 1.96
2.59
15,064
0.87
2.65
29,022
1.53
2.46
68, 743
L.MM
3.78
2.68
9,892
2.4
2.71
14,296
4.89
2.59
38,930
Table 2: Percentage of out-of-vocabulary (OOV) words in the output, normalized perplexity mea- sures (PPI) per characters and the KI. divergence between the reference and outputs of systems trained with in-domain data different language directions.
4.4.3 THE EFFECT OF DATA SIZE
Repeating the experiments in the English-to-Turkish translation direction by increasing the amount of training data with multi-domain corpora demonstrates more challenging case, where there is a greater possibility of observing new words in varying context, either in the form of morpholog- larger vocabulary extended ical inflections due to previously unobserved syntactic conditions. character-level model experi- with terminology from different domains. In this experiment, the ences a drop performance and its accuracy much lower than the subword-level one, suggesting that its capacity cannot cope with the increased amount of training data. Empirical results suggest that with increased capacity, character-level models carry the potential to reach comparable perfor- mance to subword-level models (Cherry et al. 2018). 8). On the other hand, our model reaches much larger improvement 0.82 BLEU points over the subword-level and 2.54 BLEU points over the character-level decoders, suggesting that it could make use of the increased amount of observations for improving the translation performance, which possibly aid the morphology model becoming more accurate.

---

Published as a conference paper at ICLR 2020
4.4.4 THE IMPACT OF INFLECTIONAL FEATURES
In order understand whether the latent inflectional features in fact capture information about variations related to morphological transformations, we first try generating different surface forms of the same lemma by sampling lemma vector with LMM for the input word 'go' and generating outputs using the fixed lemma vector and assigning different values to the inflectional features. In the second experiment, we assess the impact of the inflectional features by setting all features to 0 and translating set of English sentences with varying inflected forms in Turkish. Table 3 presents different sets of feature values and the corresponding outputs generated by the decoder and the outputs generated with or without the inflectional component.
Features
Output git "a git
English Translation
I.I.I.IIIIIIIII
go (informal) go at go go (formal) to go (infinitive) (hc/shc/it is) going by going (gerund) (we are) going
O,
I.I.I.I...IIIII
da git
1.0.0.1.1..O0.11...000
gidin
1.I.0.0.0.0..0...III
gitmek
0.0.1.0.0.00000000111
gidiyor
0,0.0.0.0.O.OOO.II000
gidip gidiyoruz
1000I.1110001110111100
Input he went home. he came from home. it is good to be home. his home has red walls.
Output with
Output without f
eve gitti.
eve gitti.
evden geldi.
eve geldi.
evde olmak iyi.
evde olmak iyi.
evinde kirmızi duvarlar
evde kirmızi duvar var.
Table 3: Above: Outputs of LMM based on the lemma 'git' ('go") and different sets of inflectional features. Below: Examples of predicting inflections in context with or without using features.
The model generates different surface forms for different sets of features, confirming that the latent variables represent morphological features related to the infinitive form the verb, as well as its formality conditions, prepositions, person, number and tense. Decoding the set of sentences given in the second experiment LMM always generates the correct inflectional form, although when the feature values set to 0 the model omits some inflectional features in the output, suggesting that despite partially relying on the source-side context, still encodes important information for generating correct surface forms in the inflectional features.
5 CONCLUSION
In this paper we presented novel decoding architecture for NMT employing hierarchical latent variable model to promote sparsity lexical representations, which demonstrated promising appli- cation for morphologically-rich and low-resource languages. Our model generates words one char- acter at a time by composing two latent features representing their lemmas and inflectional features. We evaluate our model against conventional open-vocabulary NMT solutions such subword and character-leved decoding methods in translationg English into three morphologically with different morphological typologies under low mid-resource settings. Our results show that our model can significantly outperform subword-level NMT models, whereas demonstrates better capacity than character-level models in coping with increased amounts of data sparsity. We also duct ablation studies on the impact of feature variations to the predictions, which prove that despite being completely unsupervised, our model can in fact manage learn morphosyntactic information and make use of it to generalize to different surface forms of words.
languages
6 ACKNOWLEDGMENTS
The authors would like to thank Marcello Federico, Orhan Firat, Adam Lopez, Graham Neubig, Akash Srivastava and Clara Vania for their feedback and suggestions. This project received funding from the European Union's Horizon 2020 research and innovation programme under grant agree- ments 825299 (GoURMET) and 6881 139199666 (SUMMA).
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Duygu Ataman, Matteo Negri, Marco Turchi, and Marcello Federico. Linguistically-mottaaae vocabulary reduction f machine translation from Turkish to English. The Prague Bulletin of Mathematical Lingnistics, 108(1):331-342 2017.
Duygu Ataman, Orhan Firat, Mattia A Di Gangi. Marcello Federico, and Alexandra Birch. On the importance of word boundaries in character-level neural machine translation. arXiv preprint arXiv:/910.066 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0477 2014.
Antonio Valerio Miceli Barone, Jindfich Helcl, Rico Sennrich, Barry Haddow, and Alexandra Birch. Deep architectures for neural machine translation. In Proceedings of the Second Conference Machine Translation, pp. 99-107, 2017.
Joost Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with differentiable binary variables. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2963-2973, 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv::308.344 2013.
Léon Bottou and Yann L.. Cun. Large scale online learning. Thrun, L. K. Saul, and B. Schölkopf (eds.), Advances in Neural Information Processing Systems 16, pp. 217-224. MIT Press, 2004.
Mohamed Boudchiche, Azzeddine Mazroui, Mohamed Ould Abdallahi Ould Bebah, Abdelhak Lakhouaja, and Abderrahim Boudlal. Alkhalil morpho sys 2: A robust arabic morpho-syntacte analyzer. Journal of King Saud Coomuuei and Information Sciences, 29(2):1411146 2017.
Mauro Cettolo. Wit3: Web inventory of transcribed and translated talks. In Conference of European Association for Machine Translation, pp. 261-268, 2012.
Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. Revisiting character-based neural machine translation with capacity and compression. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 129554305 2018.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of 8th Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST), pp. 103-111, 2014.
Jonathan H Clark, Chris Dyer, Alon Lavie, and Noah A Smith. Better hypothesis testing for statisti- cal machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 176-181. Association for Computational Linguistics, 2011.
Ryan Cotterell, Sebastian J. Mielke, Jason Eisner, and Brian Roark. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 2: Short Papers), pp. 536 536-541, 2018.
Hoang Cuong and Khalil Simaan. Latent domain translation models in mix-on-dommins haystack. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics, pp. 1928-1939, 2014.
Anirudh Goyal Alias Parth Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks. In Advances in neural information processing systems, pp. 6713-6723, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-Softmet International Conference Learning Representations, 2017.

---

Published as a conference paper at ICLR 2020
Michaell. Jordan, Zoubin Ghahramani, TommiS, Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-22 -33,, 1999.
D Kinga and Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P. Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational Bayes,
arXiv preprint
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. OpenNMT: Open- source toolkit for neural machine translation. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, System Demonstrations, pp. 67-72, 2017.
Julia Kreutzer and Artem Sokolov. Learning to segment inputs for nmt favors character-level pro- cessing. In Proceedings of the 15th International Workshop on Spoken Language Translation, pp. 166-172, 2018.
Ponnambalam Kumaraswamy, A generalized probability density function for double-bounded ran- dom processes. Journal of Hydrology, 46(1-2):79-85 1980.
Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W. Black. Character-based neural machine translation. arXiv preprint arXiv: 2015.
Pierre Lison and Jörg Tiedemann. Opensubritles2016. Extracting large parallel corpora from movie and TV subtitles. 2016.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through Lo regularization. arXiv preprint 1rXiv:1712.01312, 2018
Luong and Christopher D. Manning. Achieving open vocabulary neural machine trans- lation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1054-1063, 2016.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421, 2015.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continous re- laxation of discrete random variables. International Conference on Learning Representations,
2017.
Eric Nalisnick and Padhraic Smyth. Stick-breaking variational autoencoders. arXiv preprint arXiv:1605.06117. 2016.
Kemal Oflazer and Ilker Kuruöz. Tagging and morphological disambiguation of turkish text. In Proceedings of the fourth conference on Applied natural language processing, pp. 144-149.2 As- sociation for Computational Linguistics, 1994.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in Pytorch. NeurlPS Autodiff Workshop, 2017.
Maja Popović. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392-395, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31 International Con- ference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp- 1278-1286, 2014.
12

---

Published as a conference paper at ICLR 2020
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemat- ical Statistics, 22(3):400-40) 1951.
Gozde Gul Sahin and Mark Steedman. Character-level models versus morphology in semantic role labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 386-396, 2018.
Haşim Sak, Tunga Güngör, and Murat Saraçlar. Morphological disambiguation of turkish text with perceptron algorithm. International Conference on Intelligent Text Processing and Computa- tional Linguistics, pp. 107-118. Springer, 2007.
Philip Schulz, Wilker Aziz, and Trevor Cohn. A stochastic decoder for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1243-1252, 2018.
Rico Sennrich. How grammatical is character-level neural machine translation? Assessing MT quality with contrastive translation pairs. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2, Short Papers), pp. 376-382, 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 1715-1725, 2016.
Raivis Skadinš, Jörg Tiedemann, Roberts Rozis, and Daiga Deksne. Billions of parallel words for free: Building and using the bookshop corpus. In Proceedings the Ninth International Conference on Language Resources and Evaluation (LREC), pp. 1850-1855, 2014.
Noah A Smith. Linguistic structure prediction. Synthesis lectures on human language technologies, 4(2):1-274, 2011.
Jana Straková, Milan Straka, and Jan Hajič. Open-Source Tools for Morphology, Lemmatization, POS Tagging and Named Entity Recognition. In Proceedings of 52nd Annual Meeting of the
Association for Computational Linguisties: System Demonstrations, pp. 13 13-18. Association for Computational Linguistics, 2014.
Jörg Tiedemann. News from opus-a collection of multilingual parallel corpora with tools and inter- faces. In Recent advances in natural language processing, volume 5, pp. 237-248, 2009.
Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Proceedings of the Seventh Interna- tional Conference on Language Resources and Evaluation (LREC), pp. 2214-2218, 2012.
Francis M Tyers and Murat Serdar Alperen. South-east european times: A parallel corpus of balkan languages. In Proceedings of the LREC Workshop on Exploitation of Multilinuual Resources and Tools for Central and (South-) Eastern European Languages, pp. 49-53, 2010.
Clara Vania and Adam Lopez.. From characters words in between: Do we capture morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2016-2027, 2017.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-2)) 256, 1992.
Yonghui Wa, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. 2016.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. Variational neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 521-530, 2016.
Chunting Zhou and Graham Neubig. Multi-space variational encoder- for semi supervipe labeled sequence transduction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 310-320, 2017.
13

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1
THE STATISTICAL CHARACTERISTICS EXPERIMENTAL
Language Pair
# sentences
# tokens
# types
Source
Target
Source
Target
English-Arabic
238K
5M
4M
T20K
220K
English-Czech
118K
2M
2M
50K
118K
English-Turkish
136K
2M
3M
53K
171K
Table 4: Training sets based on the TED Talks corpora (M: Million, Thousand).
Language Pair
# sentences
tokens
types
Source
Target
Source
Target
Tnglish-Turkish
434K
SM
6M
135K
373K
Table The multi-domain training (M: Million, Thousand).
Language
Data sets
sentences
English-Arabic
Development
dev2010, test2010
6K
test2011, test2012
Testing
test2013, test2014
4K
English-Czech
Development
dev2010, test2010,
3K
test2011
Testing
test2012, test2013
3K
English- Turkish
Development
dev2010, tcst2010
3K
Testing
test2011, test2012
3K
Table 6: Development and testing sets (K: Thousand).
A.2 THE KUMARASWAMY DISTRIBUTION
KISS. 6.11
K0L0
9BS 9:51
95.8, L.I
SOLO 5.1
1.05
10
1.00
0.95
0.80
0.0
0.0
0.0
0.0
Figure 2: The top shows the density function of the continuous base distribution over (0, 1). The middle row shows the result of stretching it to include and its support. The bottom row shows the result of rectification: probability mass under (1, collapses to 0 and probability mass under r1, collapses to 1, which cause sparse outcomes to have non-zero mass. Varying the shape parameters (a, b) of the underlying continuous distribution changes how much mass concentrates outside the support (0, 1) in the stretched density, and hence the probability of sampling sparse outcomes.
14

---

Published as a conference paper at ICLR 2020
A.3 THE EFFECT OF FEATURE DIMENSIONS
We investigate the optimal lemma and inflectional feature sizes by measuring the accuracy in English-to-Turkish translation using different feature vector dimensions. The results given in Figure 3 show that gradually compressing the word representations computed by recurrent hidden states, with original dimension 512, from 500 100, leads increased output accuracy, suggesting that encoding more compact representations might provide the model with a better generalization capability. Our results also show that using a feature dimension of 10 is sufficient in reaching the best accuracy.
K-20
9
BLEU
8
7
400
Figure 3: The effect of feature dimensions on translation accuracy in Turkish.
15


Reviews:
[{"review_id": "SJl2KhcNtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of translating into morphologically-rich languages, which suffers from the problem of sparse vocabularies and high numbers of rare and unseen words. In particular, current approaches such as subwords lack explicit notions of morphology and are obtained independently of the translation objective, while operating at the character-level renders the learning of long-distance dependencies more difficult.\n\nMore concretely, this paper models the generation of target words in a stochastic, hierarchical process, where the morphological features are modelled as latent variables. At each target word, the model samples a vector representing its lemma, followed by a k-dimensional latent inflectional features (e.g. nominative or accusative). To induce sparsity in the inflectional features, the paper uses a \"stretch-and-rectify\" distribution (Louizos et al., 2018) using the Kumaraswamy distribution. The paper further applies a sparsity-inducing regulariser to encourage the inflectional features to take discrete values of \"0\" or \"1\". Parameter estimation is done by optimising a lower-bound on the marginal log-likelihood. Experiments on translations into three morphologically-rich languages, English-{Arabic, Czech, Turkish}, indicate fairly small but consistent and statistically significant gains in BLEU. Further analysis indicates improved perplexity upper bound on rare words compared to subword-based baselines, along with somewhat interpretable latent inflectional features.\n\nOverall, this paper proposes an elegant solution to an important problem, and yields statistically significant BLEU improvements over the baselines. I have listed some pros and cons below, with a recommendation of \"Weak Accept\". I would be willing to raise my scores assuming my concerns are sufficiently addressed. \n\nPros:\n1. The proposed approach is easy-to-follow and explained clearly, and the paper is overall well-written.\n\n2. Experiments are done on translation into three morphologically-rich languages with both concatenative and non-concatenative morphology, and results in small but mostly consistent BLEU improvements over the best subword, hierarchical, and character baselines for each language.\n\n3. The proposed approach is general and can potentially be applied on top of other model architectures (e.g. Transformers) and other language generation problems, such as language modelling, summarisation, etc, although the experiments in the paper focus on NMT with a GRU-based architecture.\n\nCons:\n1. It is still unclear how much computational overhead is introduced by the approach (in terms of both training and prediction times), and how scalable the approach is when applied to language pairs that have much bigger training sets. In Tables 4 and 5, it seems that the largest dataset is English-Turkish multi-domain (434K sentence pairs), which is fairly small compared to other datasets.\n\n2. Regarding the feature variations result in section 4.4.4, it is hard to draw any conclusions about the latent inflectional features just based on varying the inflectional features at one position. I would suggest running the same experiment for multiple words at different positions, and see whether the same set of inflection features always results in similar inflections. This would better convince the reader that the latent inflectional features really are capturing useful morphological information, and that the target word generation process is appropriately controlled by the latent variable.\n\n3. Since some of the experiments did not use the same standard splits (e.g. the multi-domain Turkish experiments), it would be nice to report how well external models (e.g. OpenNMT) would do in the authors' dataset split, to make sure that the reported numbers here are at least comparable to that. This would help ensure the credibility of the findings.\n\n4. The effect of data size experiments (Section 4.4.2) for the character model seems somewhat counter-intuitive. Why would making the training set bigger (by incorporating data from a more diverse domain) make the character model worse? The explanation that \"[the character model]'s capacity cannot cope with the increased amount of sparsity\" does not seem satisfactory.  Why does adding more data result in increased sparsity? Furthermore, one can simply use more hidden units or deeper layers to mitigate this problem in the character-based model. Assuming the authors are controlling for model capacity, then the character model's hidden state can be made bigger (since the vocabulary size is lower for character models, thus the embedding layer and the softmax layer are by definition smaller than the subword-based model).\n\nMinor suggestions:\n1. Section 3.5 is not very clear; adding some figures or more explanation there would help understand how prediction is done.\n\n2. It would be interesting to explore other potential metric for measuring morphological generalisation. For instance, at evaluation time, does the model predict words that it has never seen before (e.g. predicting \"plays\", even though the model has only seen \"play\" at training time) at a higher frequency than character/subword/hierarchical baselines? If yes, this would provide more evidence that the latent inflectional features are properly capturing morphological information.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1lPyVNjFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes a method, Latent Morphology Model (LMM), for producing word representations for a (hierarchical) character-level decoder used in neural machine translation (NMT). The main motivation is to overcome vocabulary sparsity or highly inflectional languages such as Arabic, Czech, and Turkish (experimented in the paper). To model the morphological inflection, they decouple lemmas and inflection types into 2 latent variables (z and f) where f is enforced to be sparse (arguably mimic the process of the human). The literature review of NMT and the discussion on the potential advantage of morphology are concise. The proposed model is a variation of Luong & Manning (2016) and Schulz et al (2018) models, thus, their main contribution is an introduction of the latent morphological features to the decoder. The proposed LMM is trained by sampling z and f from prior directly, and the sparsity of the morphological features is encouraged by L0 of the feature vector (parameterized as independent Kumaraswamy variables). They perform the main empirical study using 3 languages by translating from English to justify the proposed LMM. Lastly, they provide a quantitative analysis of the perplexities of unseen words and a qualitative on words generated of a lemma with different feature vectors.\n\nOverall, this paper could provide a novel insight into the role of modeling morphology as latent variables. However, the experiments and analyses do not sufficiently support the claim of mimicking the process of the inflection (besides the gain in performance). Some clarification would be appreciated.\n\nIn section 3, the model, to my understanding, is agnostic to the morphological inflection, except the sparsity regularization (i.e., it could model any transformation including changing the lemma itself). In addition, there is nothing in the training particularly specific about the morphology. For example, z and f are generated from only the prior whereas we could get more accurate posteriors from observing the word itself. Some background on the language might help motivate the choice of model. Are morphological inflections ambiguous? Are the morphology labels hard to obtain? I think more discussion on previous attempts to model morphology (e.g., Vylomova et al 2017, Passban et al 2018) will be very helpful to the readers.\n\nFor the experiments (section 4), the model with LMM is shown to consistently outperform character and hierarchical models. The multi-domain performance also shows the model's ability to tackle a larger vocabulary set. However, we cannot certainly conclude that the gain comes from LMM successfully modeling the inflections. A contrast of performance with less morphological languages might reveal some insight (unfortunately I do not have enough knowledge to recommend languages). Finally, the feature variation analysis is interesting, but we only see one lemma from one language. Further discussion such as the consistency of features and the types of morphology, or the similarity of the lemma vector across context, will be helpful.\n\n\nMinor Questions:\n1. Given a word, how ambiguous it is to determine the stem and the morphological type in the subject languages?\n2. How do you compute char-level PPL of the subword model?\n3. In 4.4.4, did you obtain z of `go` from just translating `go` without any context?\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkeTmSckqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to incorporate a latent model (in the form of a variational auto-encoder) in the decoding process of neural machine translation. The motivation is to capture morphology. Experiments on three language-pairs (English to Arabic, Czech, and Turkish) show promising improvements in translation accuracy (BLEU).\n\nI am quite excited by this paper. So far there are not that many successful demonstrations of VAE in neural machine translation. The method is sound and interesting. The results show convincing improvements; some practioners may argue that reported BLEU gain (e.g. 0.8) is not impressive, but I think for a new model like this it is worthy. \n\nSome suggestions or questions:\n\n- One alternative evaluation metric that might be interesting is to lemmatize both translation outputs and reference, then compute BLEU. This will help distinguish whether the proposed method is improving by getting the morphological inflections correct, or whether it is improving across the board on various word types. \n\n- Table 3 is interesting. If there is space, I would suggest more analysis along those directions, i.e. investigating what morphology is learned, what is in the latent spaces. \n\n- Do you think the results will vary depending on decoder layer depth? I wonder if different kinds of latent spaces will be learned with different depth. \n\n- Also a related question is how about varying the source input BPE merge operation? Again, it seems like these design choices might affect results, especially when dealing with morphology. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model for neural machine translation into morphologically rich languages by modeling word formation through a hierarchical latent variable model mimicking the process of morphological inflection. The model boils down to a VAE-like formulation with two latent representation: a continuous one (governed by a Gaussian) which captures lexical semantic aspects, and a discrete one (governed by the Kuma distribution) which captures the morphosyntactic function, shared among different surface forms. Even though the empirical improvements in terms of BLEU scores are fairly small, I find this a very elegant model which may foster interesting future research directions on latent models for NMT.

The reviewers had some concerns with some experimental details and model details that were properly addressed by the authors in their detailed response. In the discussion phase this alleviated the reviewers' concerns, which leads me to recommend acceptance. I urge the authors to follow the reviewer's recommendations to improve the final version of the paper. 
Decision: Accept (Spotlight)