Multi-space Variational Encoder-Decoders
for Semi-supervised Labeled Sequence Transduction
Chunting Zhou, Graham Neubig
Language Technologies Institute
Carnegie Mellon University
ctzhou,gneubig@cs.cmu.edu

Abstract
playingLabeled sequence transduction is a taskof transforming one sequence into anothersequence that satisfies desiderata specified by a set of labels. In this paper we
propose multi-space variational encoderdecoders, a new model for labeled sequence transduction with semi-supervised
learning. The generative model can use
neural networks to handle both discreteand continuous latent variables to exploit
various features of data. Experiments
show that our model provides not only apowerful supervised framework but also
can effectively take advantage of the unlabeled data. On the SIGMORPHON
morphological inflection benchmark, ourmodel outperforms single-model state-ofart results by a large margin for the majority of languages.1

1

Introduction

This paper proposes a model for labeled sequencetransduction tasks, tasks where we are given an
input sequence and a set of labels, from which
we are expected to generate an output sequencethat reflects the content of the input sequence anddesiderata specified by the labels. Several examples of these tasks exist in prior work: using labels
to moderate politeness in machine translation results (Sennrich et al., 2016), modifying the output
language of a machine translation system (Johnson et al., 2016), or controlling the length of asummary in summarization (Kikuchi et al., 2016).
In particular, however, we are motivated by thetask of morphological reinflection (Cotterell et al.,1

An implementation of our model are available
athttps://github.com/violet-zct/
MSVED-morph-reinflection.

POS=Verb, Tense=Past

playedModel

Supervised Learning

plays

Semi-Supervised Learning

Figure 1: Standard supervised labeled sequence transduction,and our proposed semi-supervised method.

2016), which we will use as an example in our description and test bed for our models.
In morphologically rich languages, different affixes (i.e. prefixes, infixes, suffixes) can be combined with the lemma to reflect various syntactic and semantic features of a word. The ability
to accurately analyze and generate morphological forms is crucial to creating applications suchas machine translation (Chahuneau et al., 2013;
Toutanova et al., 2008) or information retrieval(Darwish and Oard, 2007) in these languages. As
shown in 1, re-inflection of an inflected form given
the target linguistic labels is a challenging subtask of handling morphology as a whole, in which
we take as input an inflected form (in the example, “playing”) and labels representing the desired
form (“pos=Verb, tense=Past”) and must generate the desired form (“played”).
Approaches to this task include those utilizing
hand-crafted linguistic rules and heuristics (Tajiet al., 2016), as well as learning-based approaches
using alignment and extracted transduction rules(Durrett and DeNero, 2013; Alegria and Etxeberria, 2016; Nicolai et al., 2016). There have also
been methods proposed using neural sequenceto-sequence models (Faruqui et al., 2016; Kann
et al., 2016; Ostling, 2016), and currently ensembles of attentional encoder-decoder models (Kannand Schütze, 2016a,b) have achieved state-of-art
results on this task. One feature of these neural models however, is that they are trained in alargely supervised fashion (top of Fig. 1), using
data explicitly labeled with the input sequence andlabels, along with the output representation. Needless to say, the ability to obtain this annotated data
for many languages is limited. However, we can
expect that for most languages we can obtain large
amounts of unlabeled surface forms that may allow for semi-supervised learning over this unlabeled data (entirety of Fig. 1).2
In this work, we propose a new framework for labeled sequence transduction problems: multi-space variational encoder-decoders(MSVED, §3.3). MSVEDs employ continuousor discrete latent variables belonging to multiple
separate probability distributions3 to explain theobserved data. In the example of morphologicalreinflection, we introduce a vector of continuous
random variables that represent the lemma of thesource and target words, and also one discrete random variable for each of the labels, which are on
the source or the target side.
This model has the advantage of both providing
a powerful modeling framework for supervised
learning, and allowing for learning in an unsupervised setting. For labeled data, we maximize thevariational lower bound on the marginal log likelihood of the data and annotated labels. For unlabeled data, we train an auto-encoder to reconstruct a word conditioned on its lemma and morphological labels. While these labels are unavailable, a set of discrete latent variables are associated with each unlabeled word. Afterwards we canperform posterior inference on these latent variables and maximize the variational lower boundon the marginal log likelihood of data.
Experiments on the SIGMORPHON morphological reinflection task (Cotterell et al., 2016) findthat our model beats the state-of-the-art for a single model in the majority of languages, and is particularly effective in languages with more complicated inflectional phenomena. Further, we findthat semi-supervised learning allows for significant further gains. Finally, qualitative evaluationof lemma representations finds that our model isable to learn lemma embeddings that match withhuman intuition.
2
Faruqui et al. (2016) have attempted a limited form ofsemi-supervised learning by re-ranking with a standard ngram language model, but this is not integrated with the learning process for the neural model and gains are limited.
3
Analogous to multi-space hidden Markov models(Tokuda et al., 2002)

2

Labeled Sequence Transduction

In this section, we first present some notations regarding labeled sequence transduction problems ingeneral, then describe a particular instantiation formorphological reinflection.
Notation: Labeled sequence transduction problems involve transforming a source sequence x(s)into a target sequence x(t) , with some labels
describing the particular variety of transformation to be performed. We use discrete variables(t) (t)(t)y1 , y2 , · · · , yK to denote the labels associatedwith each target sequence, where K is the total(t) (t)(t)number of labels. Let y(t) = [y1 , y2 , · · · , yK ]denote a vector of these discrete variables. Each(t)discrete variable yk represents a categorical feature pertaining to the target sequence, and has a
set of possible labels. In the later sections, we also(t)use y(t) and yk to denote discrete latent variables
corresponding to these labels.
Given a source sequence x(s) and a set of associated target labels y(t) , our goal is to generate a target sequence x(t) that exhibits the features specified by y(t) using a probabilistic modelp(x(t) |x(s) , y(t) ). The best target sequence x̂(t) isthen given by:x̂(t) = arg max p(x(t) |x(s) , y(t) ).

(1)

x(t)

Morphological Reinflection Problem: In morphological reinflection, the source sequence x(s)consists of the characters in an inflected word(e.g., “played”), while the associated labels y(t)(t)
describe some linguistic features (e.g., ypos =(t)Verb, ytense = Past) that we hope to realizein the target. The target sequence x(t) is therefore the characters of the re-inflected form of thesource word (e.g., “played”) that satisfy the linguistic features specified by y(t) . For this task,(t)each discrete variable yk has a set of possible labels (e.g. pos=V, pos=ADJ, etc) and follows amultinomial distribution.

3

Proposed Method

3.1

Preliminaries: Variational Autoencoder

As mentioned above, our proposed model uses
probabilistic latent variables in a model basedon neural networks. The variational autoencoder(Kingma and Welling, 2014) is an efficient way
to handle (continuous) latent variables in neural

z

z

y

y

z

x

x

x

(a) VAE(b) Labeled MSVAE(c) MSVAE

z
x(s)

z

y (t)

x(t)

(d) Labeled MSVED

x(s)

y (t)

x(t)

(e) MSVED

Figure 2: Graphical models of (a) VAE, (b) labeled MSVAE,(c) MSVAE, (d) labeled MSVED, and (e) MSVED. White
circles are latent variables and shaded circles are observed
variables. Dashed lines indicate the inference process while
the solid lines indicate the generative process.

models. We describe it briefly here, and interested readers can refer to Doersch (2016) for details. The VAE learns a generative model of theprobability p(x|z) of observed data x given a latent variable z, and simultaneously uses a recognition model q(z|x) at learning time to estimate z
for a particular observation x (Fig. 2(a)). q(·) andp(·) are modeled using neural networks parameterized by φ and θ respectively, and these parameters
are learned by maximizing the variational lower
bound on the marginal log likelihood of data:log pθ (x) ≥ Ez∼qφ (z|x) [log pθ (x|z)]−KL(qφ (z|x)||p(z))(2)

The KL-divergence term (a standard feature ofvariational methods) ensures that the distributions
estimated by the recognition model qφ (z|x) do
not deviate far from our prior probability p(z) of
the values of the latent variables. To optimize
the parameters with gradient descent, Kingma
and Welling (2014) introduce a reparameterization
trick that allows for training using simple backpropagation w.r.t. the Gaussian latent variables z.
Specifically, we can express z as a deterministic
variable z = gφ (, x) where  is an independent
Gaussian noise variable  ∼ N (0, 1). The mean
µ and the variance σ 2 of z are reparameterized by
the differentiable functions w.r.t. φ. Thus, insteadof generating z from qφ (z|x), we sample the auxiliary variable  and obtain z = µφ (x) + σφ (x) ◦ ,
which enables gradients to backpropagate throughφ.
3.2

Multi-space Variational Autoencoders

As an intermediate step to our full model, we
next describe a generative model for a single sequence with both continuous and discrete latentvariables, the multi-space variational auto-encoder(MSVAE). MSVAEs are a combination of two
threads of previous work: deep generative models with both continuous/discrete latent variables
for classification problems (Kingma et al., 2014;

Maaløe et al., 2016) and VAEs with only continuous variables for sequential data (Bowman et al.,2016; Chung et al., 2015; Zhang et al., 2016;
Fabius and van Amersfoort, 2014; Bayer and Osendorfer, 2014). In MSVAEs, we have an observed sequence x, continuous latent variables z
like the VAE, as well as discrete variables y.
In the case of the morphology example, x can beinterpreted as an inflected word to be generated. yis a vector representing its linguistic labels, either
annotated by an annotator in the observed case, orunannotated in the unobserved case. z is a vectorof latent continuous variables, e.g. a latent embedding of the lemma that captures all the information
about x that is not already represented in labels y.
MSVAE: Because inflected words can be naturally
thought of as “lemma+morphological labels”, tointerpret a word, we resort to discrete and continuous latent variables that represent the linguistic labels and the lemma respectively. In this case when
the labels of the sequence y is not observed, we
perform inference over possible linguistic labels
and these inferred labels are referenced in generating x.
The generative model pθ (x, y, z)
p(z)pπ (y)pθ (x|y, z) is defined as:p(z) = N (z|0, I)Y
pπ (y) =Cat(yk |πk )

=

(3)
(4)

k

pθ (x|y, z) = f (x; y, z, θ).

(5)

Like the standard VAE, we assume the prior of
the latent variable z is a diagonal Gaussian distribution with zero mean and unit variance. We
assume that each variable in y is independent,
resulting in a factorized distribution in Eq. 4,
where Cat(yk |πk ) is a multinomial distribution
with parameters πk . For the purposes of this study,
we set these to a uniform distribution πk,j =1|πk | . f (x; y, z, θ) calculates the likelihood of x,
a function parametrized by deep neural networks.
Specifically, we employ an RNN decoder to generate the target word conditioned on the lemma variable z and linguistic labels y, detailed in §5.
When inferring the latent variables from thegiven data x, we assume the joint distribution oflatent variables z and y has a factorized form, i.e.
q(z, y|x) = q(z|x)q(y|x) as shown in Fig. 2(c).

3.3

The inference model is defined as follows:qφ (z|x) = N (z|µφ (x), diag(σφ2 (x)))
Yqφ (y|x) =qφ (yk |x)

(6)

k

=

Y

Cat(yk |πφ (x))(7)

kwhere the inference distribution over z is a diagonal Gaussian distribution with mean and varianceparameterized by neural networks. The inference
model q(y|x) on labels y has the form of a discriminative classifier that generates a set of multinomial probability vectors πφ (x) over all labels
for each tag yk . We represent each multinomial
distribution q(yk |x) with an MLP.
The MSVAE is trained by maximizing the following variational lower bound U(x) on the objective for unlabeled data:log pθ (x) ≥ E(y,z)∼qφ (y,z|x) logpθ (x, y, z)
qφ (y, z|x)

= Ey∼qφ (y|x) [Ez∼qφ (z|x) [log pθ (x|z, y)]− KL(qφ (z|x)||p(z)) + log pπ (y)− log qφ (y|x)] = U(x)(8)

Note that this introduction of discrete variables
requires more sophisticated optimization algorithms, which we will discuss in §4.1.
Labeled MSVAE: When y is observed as shownin Fig. 2(b), we maximize the following variational lower bound on the marginal log likelihoodof the data and the labels:
log pθ (x, y) ≥ Ez∼qφ (z|x) logpθ (x, y, z)=
qφ (z|x)(9)

which is a simple extension to Eq. 2.
Note that when labels are not observed, the inference model qφ (y|x) has the form of a discriminative classifier, thus we can use observed labelsas the supervision signal to learn a better classifier.
In this case we also minimize the following crossentropy as the classification loss:D(x, y) = E(x,y)∼pl (x,y) [− log qφ (y|x)]

Finally, we discuss the full proposed method:
the multi-space variational encoder-decoder(MSVED), which generates the target x(t) fromthe source x(s) and labels y(t) . Again, we discuss
two cases of this model: labels of the target
sequence are observed and not observed.
MSVED: The graphical model for the MSVED isgiven in Fig. 2 (e). Because the labels of target sequence are not observed, once again we treat themas discrete latent variables and make inference
on the these labels conditioned on the target sequence. The generative process for the MSVED isvery similar to that of the MSVAE with one important exception: while the standard MSVAE conditions the recognition model q(z|x) on x, then generates x itself, the MSVED conditions the recognition model q(z|x(s) ) on the source x(s) , then generates the target x(t) . Because only the recognition model is changed, the generative equations forpθ (x(t) , y(t) , z) are exactly the same as Eqs. 3–5
with x(t) swapped for x and y(t) swapped for y.
The variational lower bound on the conditional log
likelihood, however, is affected by the recognition
model, and thus is computed as:log pθ (x(t) |x(s) )≥E(y(t) ,z)∼qφ (y(t) ,z|x(s) ,x(t) ) logpθ (x(t) , y(t) , z|x(s) )qφ (y(t) , z|x(s) , x(t) )=Ey(t) ∼qφ (y(t) |x(t) ) [Ez∼qφ (z|x(s) ) [log pθ (x(t) |y(t) , z)]− KL(qφ (z|x(s) )||p(z)) + log pπ (y(t) )− log qφ (y(t) |x(t) )] = Lu (x(t) |x(s) )Ez∼qφ (z|x) [log pθ (x|y, z) + log pπ (y)]− KL(qφ (z|x)||p(z))

Multi-space Variational
Encoder-Decoders

Labeled MSVED: When the complete form ofx(s) , y(t) , and x(t) is observed in our training data,
the graphical model of the labeled MSVED model
is illustrated in Fig. 2 (d). We maximize the variational lower bound on the conditional log likelihood of observing x(t) and y(t) as follows:
log pθ (x(t) , y(t) |x(s) )≥ Ez∼qφ (z|x(s) ) logpθ (x(t) , y(t) , z|x(s) )qφ (z|x(s) )= Ez∼qφ (z|x(s) ) [log pθ (x(t) |y(t) , z) + log pπ (y(t) )]−(10)

where pl (x, y) is the distribution of labeled data.
This is a form of multi-task learning, as this additional loss also informs the learning of our representations.

(11)

KL(qφ (z|x(s) )||p(z)) = Ll (x(t) , y(t) |x(s) ) (12)4Learning MSVED

Now that we have described our overall model, we
discuss details of the learning process that proveuseful to its success.
4.1

Learning Discrete Latent Variables

One challenge in training our model is that it is nottrivial to perform back-propagation through discrete random variables, and thus it is difficult tolearn in the models containing discrete tags suchas MSVAE or MSVED.4 To alleviate this problem,
we use the recently proposed Gumbel-Softmax
trick (Maddison et al., 2014; Gumbel and Lieblein,
1954) to create a differentiable estimator for categorical variables.
The Gumbel-Max trick (Gumbel and Lieblein,
1954) offers a simple way to draw samples froma categorical distribution with class probabilities π1 , π2 , · · · by using the argmax operation as
follows: one hot(arg maxi [gi + log πi ]), where
g1 , g2 , · · · are i.i.d. samples drawn from the Gumbel(0,1) distribution.5 When making inferences on
the morphological labels y1 , y2 , · · · , the GumbelMax trick can be approximated by the continuous
softmax function with temperature τ to generate asample vector ŷi for each label i:exp((log(πij ) + gij )/τ )ŷij = PNi
k=1 exp((log(πik ) + gik )/τ

(13)

where Ni is the number of classes of label i. Whenτ approaches zero, the generated sample ŷi becomes a one-hot vector. When τ > 0, ŷi is smooth
w.r.t πi . In experiments, we start with a relatively
large temperature and decrease it gradually.
4.2

this case, the RNN decoder can easily rely on thetrue output of last time step during training to decode the next token, which degenerates into an
RNN language model. Hence, the latent variables
are ignored by the decoder and cannot encode any
useful information. The latent variable z learns anundesirable distribution that coincides with the imposed prior distribution but has no contribution to
the decoder. To force the decoder to use the latentvariables, we take the following two approaches
which are similar to Bowman et al. (2016).
KL-Divergence Annealing: We add a coefficient
λ to the KL cost and gradually anneal it from zero
to a predefined threshold λm . At the early stageof training, we set λ to be zero and let the model
first figure out how to project the representationof the source sequence to a roughly right point in
the space and then regularize it with the KL cost.
Although we are not optimizing the tight variational lower bound, the model balances well between generation and regularization. This technique can also be seen in (Kočiskỳ et al., 2016;
Miao and Blunsom, 2016).
Input Dropout in the Decoder: Besides annealing the KL cost, we also randomly drop out the
input token with a probability of β at each time
step of the decoder during learning. The previous
ground-truth token embedding is replaced with azero vector when dropped. In this way, the RNN
decoder could not fully rely on the ground-truth
previous token, which ensures that the decoder
uses information encoded in the latent variables.

Learning Continuous Latent Variables

MSVED aims at generating the target sequence
conditioned on the latent variable z and the target labels y(t) . This requires the encoder togenerate an informative representation z encoding the content of the x(s) . However, the variational lower bound in our loss function contains
the KL-divergence between the approximate posterior qφ (z|x) and the prior p(z), which is relatively easy to learn compared with learning to
generate output from a latent representation. We
observe that with the vanilla implementation the
KL cost quickly decreases to near zero, setting
qφ (z|x) equal to standard normal distribution. In
4
Kingma et al. (2014) solve this problem by marginalizing over all labels, but this is infeasible in our case where wehave an exponential number of label combinations.
5
The Gumbel (0,1) distribution can be sampled byfirst drawing u ∼ Uniform(0,1) and computing g =− log(− log(u)).

5Architecture for Morphological
Reinflection

Training details: For the morphological reinflection task, our supervised training data consists ofsource x(s) , target x(t) , and target tags y(t) . We
test three variants of our model trained using different types of data and different loss functions.
First, the single-directional supervised model (SDSup) is purely supervised: it only decodes thetarget word from the given source word with theloss function Ll (x(t) , y(t) |x(s) ) from Eq. 12. Second, the bi-directional supervised model (BDSup) is trained in both directions: decoding thetarget word from the source word and decoding
the source word from the target word, which corresponds to the loss function Ll (x(t) , y(t) |x(s) ) +Lu (x(s) |x(t) ) using Eqs. 11–12. Finally, the semisupervised model (Semi-sup) is trained to maxi-

Language

#LD

#ULD

Proposed MSVED
SD-Sup BD-Sup Semi-sup

Baseline MED
SingleEnsemble

TurkishHungarian
SpanishRussian
NavajoMalteseArabicGeorgianGermanFinnish12,79819,20012,79912,798
12,635
19,20012,797
12,795
12,777
12,80029,60834,025
72,151
67,691
6,839
46,91853,791
46,562
56,246
74,687

93.25
97.00
88.32
75.77
85.00
84.83
79.13
89.31
75.55
75.59

95.66†
98.54†
91.50
83.07
95.37†
88.21†
92.62†
93.63†
84.08
85.11

97.25‡
99.16‡
93.74
86.80‡
98.25‡
88.46‡
93.37‡
95.97‡
90.28‡
91.20‡

89.56
96.46
‡
94.74†
†
83.55
63.62
79.59
72.58
91.06
89.11†
85.63†

95.00
98.37
96.69
87.13
83.00
84.25
82.80
96.21
92.41
93.18

Avg. Acc

–

–

84.38

90.78†

93.45‡

84.59

90.90

Table 1: Results for Task 3 of SIGMORPHON 2016 on Morphology Reinflection. † represents the best single supervised
model score, ‡ represents the best model including semi-supervised models, and bold represents the best score overall. #LD
and #ULD are the number of supervised data and unlabeled words respectively.
✏ ⇠ N (0, 1)

Source Word

⌃(x)

ä

k

Reinflected Form......

z

µ(x)

k

<w>

+
k

a

b

l

y1T

y2T

y3T

y4T ....

Supervised Variational Encoder Decoder✏ ⇠ N (0, 1) k

Source Word

⌃(x)

µ(x)

a
Source Word......

zk<w>
Multinomial Samplingy1 2 {pos: V, N, ADJ}..

y1
k

a

l

b

y2
y3
y4
···

y2 2 {def: DEF, INDEF}+

y3 2 {num: DU, SG, PL}...
...
...

Unsupervised Variational Auto-encoder

Figure 3: Model architecture for labeled and unlabeled data.
For the encoder-decoder model, only one direction from thesource to target is given. The classification model is not illustrated in the diagram.

mize the variational lower bounds and minimize
the classification cross-entropy error of 10.
L(x(s) , x(t) , y(t) , x) = α · U(x) + Lu (x(s) |x(t) )+ Ll (x(t) , y(t) |x(s) ) − D(x(t) , y(t) )(14)

The weight α controls the relative weight between
the loss from unlabeled data and labeled data.
We use Monte Carlo methods to estimate theexpectation over the posterior distribution q(z|x)
and q(y|x) inside the objective function 14.
Specifically, we draw Gumbel noise and Gaussian
noise one at a time to compute the latent variables
y and z.
The overall model architecture is shown in
Fig. 3. Each character and each label is associated with a continuous vector. We employ Gated
Recurrent Units (GRUs) for the encoder and de-→−←−coder. Let ht and ht denote the hidden state of theforward and backward encoder RNN at time stept. u is the hidden representation of x(s) concatenating the last hidden state from both directions−→ ←−i.e. [hT ; hT ] where T is the word length. u isused as the input for the inference model on z. We
represent µ(u) and σ 2 (u) as MLPs and sample z
from N (µ(u), diag(σ 2 (u))), using z = µ + σ ◦ ,
where  ∼ N (0, I). Similarly, we can obtain thehidden representation of x(t) and use this as input(t)
to the inference model on each label yi which isalso an MLP following a softmax layer to generate
the categorical probabilities of target labels.
In decoding, we use 3 types of information in
calculating the probability of the next character :(1) the current decoder state, (2) a tag context vector using attention (Bahdanau et al., 2015) over
the tag embeddings, and (3) the latent variable z.
The intuition behind this design is that we would
like the model to constantly consider the lemma
represented by z, and also reference the tag corresponding to the current morpheme being generated at this point. We do not marginalize overthe latent variable z however, instead we use themode µ of z as the latent representation for z. We
use beam search with a beam size of 8 to perform
search over the character vocabulary at each decoding time step.
Other experimental setups: All hyperparameters are tuned on the validation set, and include
the following: For KL cost annealing, λm is set
to be 0.2 for all language settings. For character
drop-out at the decoder, we empirically set β to
be 0.4 for all languages. We set the dimension ofcharacter embeddings to be 300, tag label embeddings to be 200, RNN hidden state to be 256, andlatent variable z to be 150. We set α the weight
for the unsupervised loss to be 0.8. We train themodel with Adadelta (Zeiler, 2012) and use earlystop with a patience of 10.

Experiments

6.1

Background: SIGMORPHON 2016

SIGMORPHON 2016 is a shared task on morphological inflection over 10 different morphologically rich languages. There are a total of threetasks, the most difficult of which is task 3, which
requires the system to output the reinflection of an
inflected word.6 The training data format in task
3 is in triples: (source word, target labels, target
word). In the test phase, the system is asked togenerate the target word given a source word andthe target labels. There are a total of three tracks
for each task, divided based the amount of supervised data that can be used to solve the problem,
among which track 2 has the strictest limitation of
only using data for the corresponding task. As thisis an ideal testbed for our method, which can learn
from unlabeled data, we choose track 2 and task 3
to test our our model’s ability to exploit this data.
As a baseline, we compare our results with the
MED system (Kann and Schütze, 2016a) which
achieved state-of-the-art results in the shared task.
This system used an encoder-decoder model withattention on the concatenated source word and target labels. Its best result is obtained from an ensemble of five RNN encoder-decoders (Ensemble). To make a fair comparison with our models, which don’t use ensembling, we also calculated single model results (Single).
All models are trained using the labeled training
data provided for task 3. For our semi-supervised
model (Semi-sup), we also leverage unlabeleddata from the training and validation data for tasks
1 and 2 to train variational auto-encoders.
6.2

Results and Analysis

From the results in Tab. 1, we can glean a numberof observations. First, comparing the results of ourfull Semi-sup model, we can see that for all languages except Spanish, it achieves accuracies better than the single MED system, often by a large
margin. Even compared to the MED ensembled
model, our single-model system is quite competitive, achieving higher accuracies for Hungarian,
6
Task 1 is inflection of a lemma word and task 2 is reinflection but also provides the source word labels.

PrefixStem

Suffix

TurkishHungarian
SpanishRussian
NavajoMalteseArabicGeorgianGermanFinnish

0.21
0.00
0.09
0.66
77.64
48.81
68.52
4.46
0.84
0.02

1.12
0.08
3.25
7.70
18.38
11.05
37.04
0.41
3.32
12.33

98.75
99.79
90.74
85.00
26.40
98.74
88.24
92.47
89.19
96.16

Table 2: Percentage of inflected word forms that have modified each part of the lemma (Cotterell et al., 2016) (somewords can be inflected zero or multiple times, thus sums may
not add to 100%).

Accuracy6

Language100
95 Navajo
90
85
80
75
70
65
600.2
0.3

ArabicMalteseHungarianTurkishGeorgian
Finnish Spanish
GermanRussianMSVED
MED (1)

0.4
0.5
0.6
0.7
0.8
Percentage of suffixing inflection

0.9

1.0

Figure 4: Performance on test data w.r.t. the percentage ofsuffixing inflection. Points with the same x-axis value correspond to the same language results.

Navajo, Maltese, and Arabic, as well as achieving
average accuracies that are state-of-the-art.
Next, comparing the different varieties of our
proposed models, we can see that the semisupervised model consistently outperforms thebidirectional model for all languages. And similarly, the bidirectional model consistently outperforms the single direction model. From these results, we can conclude that the unlabeled data isbeneficial to learn useful latent variables that canbe used to decode the corresponding word.
Examining the linguistic characteristics of themodels in which our model performs well provides even more interesting insights. Cotterell
et al. (2016) estimate how often the inflection
process involves prefix changes, stem-internalchanges or suffix changes, the results of which
are shown in Tab. 2. Among the many languages,
the inflection processes of Arabic, Maltese andNavajo are relatively diverse, and contain a large
amount of all three forms of inflection. By examining the experimental results together with themorphological inflection process of different languages, we found that among all the languages,
Navajo, Maltese and Arabic obtain the largest
gains in performance compared with the ensem-

a l - - i mā r ā t i y y ā t ucase=NOM
mood=None
pos=ADJn ı́ d a j i d l e e hmood=REAL
0.8
pos=V

per=None
poss=None

per=4

0.6

num=PLtense=None
aspect=Nonevoice=None
gen=FEM
def=DEF

num=PL

0.4

aspect=
IPFV/PROG

0.2

arg=None
0.0

Figure 5: Two examples of attention weights on target linguistic labels: Arabic (Left) and Navajo (Right). When a tag
equals None, it means the word does not have this tag.

bled MED system. To demonstrate this visually, inFig. 4, we compare the semi-supervised MSVED
with the MED single model w.r.t. the percentage
of suffixing inflection of each language, showing
this clear trend.
This strongly demonstrates that our model is agnostic to different morphological inflection forms
whereas the conventional encoder-decoder withattention on the source input tends to perform better on suffixing-oriented morphological inflection.
We hypothesize that for languages that the inflection mostly comes from suffixing, transduction isrelatively easy because the source and target words
share the same prefix and the decoder can copy theprefix of the source word via attention. However,for languages in which different inflections of alemma go through different morphological processes, the inflected word and the target word may
differ greatly and thus it is crucial to first analyze
the lemma of the inflected word before generating the corresponding the reinflection form basedon the target labels. This is precisely what ourmodel does by extracting the lemma representation z learned by the variational inference model.

6.3

Analysis on Tag Attention

To analyze how the decoder attends to the linguistic labels associated with the target word, we randomly pick two words from the Arabic and Navajo
test set and plot the attention weight in Fig. 5.
The Arabic word “al-’imārātiyyātu” is an adjective which means “Emirati”, and its source word in
the test data is “’imārātiyyin” 7 . Both of these are
declensions of “’imārātiyy”. The source word is7
https://en.wiktionary.org/wiki/%D8%
A5%D9%85%D8%A7%D8%B1%D8%A7%D8%AA%D9%8A

Figure 6: Visualization of latent variables z for Maltese with35 pseudo-lemma groups in the figure.

singular, masculine, genitive and indefinite, while
the required inflection is plural, feminine, nominative and definite. We can see from the left heat
map that the attention weights are turned on at several positions of the word when generating corresponding inflections. For example, “al-” in Arabic is the definite article that marks definite nouns.
The same phenomenon can also be observed in the
Navajo example, as well as other languages, but
due to space limitation, we don’t provide detailed
analysis here.
6.4

Visualization of Latent Lemmas

To investigate the learned latent representations,in this section we visualize the z vectors, examining whether the latent space groups together
words with the same lemma. Each sample in SIGMORPHON 2016 contains source word and target words which share the same lemma. We run aheuristic process to assign pairs of words to groups
that likely share a lemma by grouping together
word pairs for which at least one of the words ineach pair shares a surface form. This process is
not error free – errors may occur in the case wheremultiple lemmas share the same surface form – butin general the groupings will generally reflect lemmas except in these rare erroneous cases, so we
dub each of these groups a pseudo-lemma.
In Fig. 6, we randomly pick 1500 words from
Maltese and visualize the continuous latent vectors of these words. We compute the latent vectors as µφ (x) in the variational posterior inference(Eq. 6) without adding the variance. As expected,
words that belong to the same pseudo-lemma (inthe same color) are projected into adjacent points
in the two-dimensional space. This demonstrates
that the continuous latent variable captures the
canonical form of a set of words and demonstrates
the effectiveness of the proposed representation.

LanguageSrc Word

Tgt Labels

Gold Tgt

MED

Ours

Turkishkocamayaratmamdan
bitimizde

pos=N,poss=PSS1S,case=ESS,num=SG
pos=N,case=NOM,num=SG
pos=N,tense=PST,per=1,num=SGkocamdayaratma
bittikkocamayaratmabitiydik

kocamdayaratman
bittimMaltesendammhomlitqożżhieli
tissikkmuhomli

pos=V,polar=NEG,tense=PST,num=SG
pos=V,polar=NEG,tense=PST,num=SG
pos=V,polar=POS,tense=PST,num=PL

tindammhiex
tqożżxssikkmulnandammejthiex
tqożżx
tissikkmulna

tindammhiex
qażżejtx
tissikkmulna

Finnishverovapaatta
turrumme
sukunimin

pos=ADJ,case=PRT,num=PL
pos=V,mood=POT,tense=PRS,num=PL
pos=N,case=PRIV,num=PL

verovapaita
turtunemme
sukunimitt

verovappaita
turtunemme
sukunimeitta

verovapaita
turrunemme
sukunimeitta

6.6

Case Study on Reinflected Words

In Tab. 3, we examine some model outputs on thetest data from the MED system and our model respectively. It can be seen that most errors of MED
and our models can be ascribed to either over-copy
or under-copy of characters. In particular, from thecomplete outputs we observe that our model tends
to be more aggressive in its changes, resulting in# Unlabeled words

>15e5

5e4

3e4

2e4

From Tab. 1, we can see that semi-supervised
learning always performs better than supervised
learning without unlabeled data. In this section,
we investigate to what extent the size of unlabeleddata can help with performance. We process a
German corpus from a 2017 Wikipedia dump andobtain more than 100,000 German words. Thesewords are ranked in order of occurrence frequency
in Wikipedia. The data contains a certain amountof noise since we did not apply any special processing. We shuffle all unlabeled data from boththe Wikipedia and the data provided in the sharedtask used in previous experiments, and increase
the number of unlabeled words used in learningby 10,000 each time, and finally use all the unlabeled data (more than 150,000 words) to train
the model. Fig. 7 shows that the performanceon the test data improves as the amount of unlabeled data increases, which implies that the unsupervised learning continues to help improve themodel’s ability to model the latent lemma representation even as we scale to a noisy, real, and relatively large-scale dataset. Note that the growth rateof the performance grows slower as more data isadded, because although the number of unlabeleddata is increasing, the model has seen most word
patterns in a relatively small vocabulary.

92
91
90
89
88
87
86
85
84
83

1e4

Analyzing Effects of Size of Unlabeled
Data

0

6.5

Accuracy on test data (%)

Table 3: Randomly picked output examples on the test data. Within each block, the first, second and third lines are outputs that
ours is correct and MED’s is wrong, ours is wrong and MED’s is correct, both are wrong respectively.

Figure 7: Performance on the German test data w.r.t. theamount of unlabeled Wikipedia data.

it performing more complicated transformations,both successfully (such as Maltese “ndammhomli”
to “tindammhiex”) and unsuccessfully (“tqożżx”
to “qażżejtx”). In contrast, the attentional encoderdecoder model is more conservative in its changes,
likely because it is less effective in learning an abstracted representation for the lemma, and insteadcopies characters directly from the input.

7

Conclusion and Future WorkIn this work, we propose a multi-space variational
encoder-decoder framework for labeled sequencetransduction problem. The MSVED performs wellin the task of morphological reinflection, outperforming the state of the art, and further improving with the addition of external unlabeled data.
Future work will adapt this framework to othersequence transduction scenarios such as machinetranslation, dialogue generation, question answering, where continuous and discrete latent variables
can be abstracted to guide sequence generation.

Acknowledgments
The authors thank Jiatao Gu, Xuezhe Ma, Zihang
Dai and Pengcheng Yin for their helpful discussions. This work has been supported in part by an
Amazon Academic Research Award.

ReferencesIñaki Alegria and Izaskun Etxeberria. 2016. Ehu at
the sigmorphon 2016 shared task. a simple proposal:
Grapheme-to-phoneme for inflection. In Proceedings of the 2016 Meeting of SIGMORPHON .
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. The International
Conference on Learning Representations .
Justin Bayer and Christian Osendorfer. 2014. Learning stochastic recurrent networks. arXiv preprint
arXiv:1411.7610 .
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous
space. Proceedings of CoNLL .
Victor Chahuneau, Eva Schlinger, Noah A Smith, andChris Dyer. 2013. Translating into morphologicallyrich languages with synthetic phrases. Association
for Computational Linguistics.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth
Goel, Aaron C Courville, and Yoshua Bengio. 2015.
A recurrent latent variable model for sequential data.
In Advances in neural information processing systems. pages 2980–2988.
R.

Cotterell, C. Kirov, J. Sylak-Glassman,
D. Yarowsky, J. Eisner, and M. Hulden. 2016.
The sigmorphon 2016 shared taskmorphologicalreinflection. In Proceedings of the 54th Annual
Meeting of the Association for Computational
Linguistics.

Kareem Darwish and Douglas W Oard. 2007. Adapting morphology for arabic information retrieval. In
Arabic Computational Morphology, Springer, pages
245–262.
Carl Doersch. 2016. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908 .
Greg Durrett and John DeNero. 2013. Supervised
learning of complete morphological paradigms. In
Proceedings of the 2013 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Association for Computational Linguistics, Atlanta,
Georgia, pages 1185–1195.
Otto Fabius and Joost R van Amersfoort. 2014. Variational recurrent auto-encoders. arXiv preprint
arXiv:1412.6581 .
Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, andChris Dyer. 2016. Morphological inflection generation using character sequence to sequence learning. In Proceedings of the 2016 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pages 634–643.

Emil Julius Gumbel and Julius Lieblein. 1954. Statistical theory of extreme values and some practicalapplications: a series of lectures. US Government
Printing Office Washington .
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim
Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat,
Fernanda Viégas, Martin Wattenberg, Greg Corrado,
et al. 2016. Google’s multilingual neural machinetranslation system: Enabling zero-shot translation.
arXiv preprint arXiv:1611.04558 .
Katharina Kann, Ryan Cotterell, and Hinrich Schütze.
2016. Neural multi-source morphological reinflection. arXiv preprint arXiv:1612.06027 .
Katharina Kann and Hinrich Schütze. 2016a. Med:The lmu system for the sigmorphon 2016 shared task
on morphological reinflection. In In Proceedingsof the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Berlin, Germany.
Katharina Kann and Hinrich Schütze. 2016b. Singlemodel encoder-decoder with explicit morphologicalrepresentation for reinflection. In In Proceedings of
the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany.
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya
Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In
Proceedings of the 2016 Conference on Empirical
Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas,
pages 1328–1338.
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez
Rezende, and Max Welling. 2014. Semi-supervised
learning with deep generative models. In Advances in Neural Information Processing Systems.
Montréal, Canada, pages 3581–3589.
D.P. Kingma and M. Welling. 2014. Auto-encoding
variational bayes. In The International Conference
on Learning Representations.
Tomáš Kočiskỳ, Gábor Melis, Edward Grefenstette,
Chris Dyer, Wang Ling, Phil Blunsom, andKarl Moritz Hermann. 2016. Semantic parsing withsemi-supervised sequential autoencoders. the 2016
Conference on Empirical Methods in Natural Language Processing (EMNLP) .
Lars Maaløe, Casper Kaae Sønderby, Søren Kaae
Sønderby, and Ole Winther. 2016. Auxiliary deep
generative models. Proceedings of the 33rd International Conference on Machine Learning .
Chris J Maddison, Daniel Tarlow, and Tom Minka.
2014. A* sampling. In Advances in Neural Information Processing Systems. pages 3086–3094.

Yishu Miao and Phil Blunsom. 2016. Language as alatent variable: Discrete generative models for sentence compression. the 2016 Conference on Empirical Methods in Natural Language Processing(EMNLP) .
Garrett Nicolai, Bradley Hauer, Adam St. Arnaud, andGrzegorz Kondrak. 2016. Morphological reinflection via discriminative string transduction. In Proceedings of the 2016 Meeting of SIGMORPHON .
Robert Ostling. 2016. Morphological reinflection withconvolutional neural networks. In Proceedings ofthe 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology page 23.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Controlling politeness in neural machinetranslation via side constraints. In Proceedings ofthe 2016 Conference of The North American Chapter of the Association for Computational Linguistics(NAACL). pages 35–40.
Dima Taji, Ramy Eskander, Nizar Habash, and Owen
Rambow. 2016. The columbia university - new york
university abu dhabi sigmorphon 2016 morphological reinflection shared task submission. In Proceedings of the 2016 Meeting of SIGMORPHON .
Keiichi Tokuda, Takashi Masuko, Noboru Miyazaki,
and Takao Kobayashi. 2002. Multi-space probability distribution hmm. IEICE TRANSACTIONS on
Information and Systems 85(3):455–464.
Kristina Toutanova, Hisami Suzuki, and Achim Ruopp.
2008. Applying morphology generation models tomachine translation. In Proceedings of the 46th Annual Meeting of the Association for Computational
Linguistics. pages 514–522.
Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 .
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and
Min Zhang. 2016. Variational neural machine translation. Proceedings of the 54th Annual Meeting of
the Association for Computational Linguistics .