Morphological Inflection Generation with Multi-space Variational
Encoder-Decoders
Chunting Zhou and Graham Neubig
Language Technologies Institute
Carnegie Mellon University
ctzhou,gneubig@cs.cmu.edu

Abstract
This paper describes the CMU submission to shared task 1 of SIGMORPHON
2017.
The system is based on themulti-space variational encoder-decoder(MSVED) method of Zhou and Neubig(2017), which employs both continuousand discrete latent variables for the variational encoder-decoder and is trained in asemi-supervised fashion. We discuss somelanguage-specific errors and present result
analysis.

1

Introduction

In morphologically rich languages, different affixes (i.e. prefixes, infixes, suffixes) can be combined with the lemma to reflect various syntactic
and semantic features of a word. In many areasof natural language processing (NLP) it is important that systems are able to correctly analyze andgenerate different morphological forms, including
previously unseen forms. The ability to accurately
analyze and generate morphological forms is crucial to creating applications such as machine translation (Chahuneau et al., 2013) and information
retrieval (Darwish and Oard, 2007). Accordingly,
learning morphological reinflection patterns fromlabeled data is an important challenge.
The Universal Morphological Reinflection task
at SIGMORPHON 2017 (Cotterell and Schütze,2017) is an evaluation campaign aimed at systems
that tackle the task of morphological inflection. It
extends the SIGMORPHON 2016 Morphological
Reinflection by conducting tasks in 52 languages
instead of 10 Cotterell et al. (2016).
In our system submission, we utilize multispace variational encoder-decoders (MSVEDs),
which are a varitional encoder-decoder with both
continuous and discrete latent variables (Zhou andNeubig, 2017). The continuous latent variable
is expected to reflect the lemma form of a word
and the discrete variables are used to induce thedesired labels of the inflected word. The wholemodel is trained in a semi-supervised fashion. For
the supervised part we are reducing the reconstruction error of generating the inflected word
given the lemma and corresponding tags. For theunsupervised part, we introduce the discrete latent variables representing the morphological tags,
and train an auto-encoder over unlabeled corpora.
Thus, the training objective includes both the variational lower bound on the marginal log likelihood of the observed parallel training data and themonolingual data.
There are two tasks in SIGMORPHON 2017,
which are morphology inflection (task 1) andparadigm completion (task 2) respectively. We
participated in task 1, inflection generation, in
which the goal is to output the inflected form of alemma given a set of desired morphological tags.1
Experimental results found that our model works
relatively well on the shared task 1 without extensive tuning of hyper-parameters and languagespecific features.

2

Methods

In this section we will detail the multi-space variational encoder-decoder model.
Notation:
In morphological reinflection, thesource sequence x(s) consists of the characters in
an inflected word (e.g., “played”), while the associated labels y(t) describe some linguistic fea(t)(t)
tures (e.g., ypos = Verb, ytense = Past) that we
1
We considered participation in task 2, but while the training data in the second task provides all inflection forms foreach lemma, the number of different lemmas is rather smaller,
which resulted in our model quickly overfitting to the training
data when training the neural model. Therefore, we only took
part in the first task this time.

hope to realize in the target. The target sequencex(t) is therefore the characters of the re-inflected
form of the source word (e.g., “played”) that satisfy the linguistic features specified by y(t) . For(t)this task, each discrete variable yk has a set of
possible labels (e.g. pos=V, pos=ADJ, etc)
and follows a multinomial distribution.
2.1

Preliminaries: Variational Autoencoder

The variational autoencoder (Kingma andWelling, 2014) is an efficient way to handle(continuous) latent variables in neural models.
We describe it briefly here, and interested readers
can refer to Doersch (2016) for details. The
VAE learns a generative model of the probabilityp(x) of observed data x. The generative process
consists of first generating a continuous latentvariable z conditioned on the observed datax, which is termed as the recognition modelq(z|x) (encoder) and then use this latent variable
to reconstruct the observation x known as thereconstruction (decoder) model p(x|z). VAE
uses the variational inference to approximate theintractable posterior by learning a parametricposterior distribution for all observations.Th
learning objective function is the variational lower
bound on the marginal log likelihood of data:log pθ (x(t) , y(t) |x(s) )≥ Ez∼qφ (z|x(s) ) logpθ (x(t) , y(t) , z|x(s) )qφ (z|x(s) )= Ez∼qφ (z|x(s) ) [log pθ (x(t) |y(t) , z) + log pπ (y(t) )]−KL(qφ (z|x(s) )||p(z)) = Ll (x(t) , y(t) |x(s) )which is a simple extension to the vanilla variational auto-enocders.
Second, in the case of unsupervised learning or
when the labels of the inflected word is not observed, we only observe a word or a pair of words
and we would like to maximize the log likelihoodof the observed data by marginalizing over possible morphological labels, which is consisted to thesupervised case above. In this scenario, we canintroduce the discrete latent variables for the inflected labels which are used to infer the labels forthe target word. Then when decoding the word, we
condition both on the continuous and discrete latent variables. For the variational encoder-decoder(MSVED), the variational lower bound on the conditional log likelihood is affected by the recognition model, and thus is computed as:log pθ (x(t) |x(s) )≥E(y(t) ,z)∼qφ (y(t) ,z|x(s) ,x(t) ) logpθ (x(t) , y(t) , z|x(s) )qφ (y(t) , z|x(s) , x(t) )(1)− KL(qφ (z|x(s) )||p(z)) + log pπ (y(t) )− log qφ (y(t) |x(t) )] = Lu (x(t) |x(s) )To optimize the parameters with gradient descent,Kingma and Welling (2014) introduce a reparameterization trick that allows for training using simple backpropagation w.r.t. the Gaussian latent variables z.
2.2

(2)

=Ey(t) ∼qφ (y(t) |x(t) ) [Ez∼qφ (z|x(s) ) [log pθ (x(t) |y(t) , z)]log pθ (x) ≥ Ez∼qφ (z|x) [log pθ (x|z)]−KL(qφ (z|x)||p(z))

follows:

Multi-space Variational
Encoder-Decoders

There are two cases to discuss when employing
the variational encoder-decoder framework for labeled sequence transduction. First, when the labels of the inflected words are known as is theformat of the training data in the shared task, we
don’t need to bother introduction the discrete latent variables for the inflected labels. We maximize the variational lower bound on the conditional log likelihood of observing x(t) and y(t) as(3)

While the unsupervised objective is trained by
maximizing the following variational lower boundU(x) on the objective for unlabeled data:log pθ (x) ≥ E(y,z)∼qφ (y,z|x) logpθ (x, y, z)
qφ (y, z|x)

= Ey∼qφ (y|x) [Ez∼qφ (z|x) [log pθ (x|z, y)]− KL(qφ (z|x)||p(z)) + log pπ (y)− log qφ (y|x)] = U(x)(4)

Note that when labels are not observed, the inference model qφ (y|x) has the form of a discriminative classifier, thus we can use observed labelsas the supervision signal to learn a better classifier.
In this case we also minimize the following crossentropy as the classification loss:D(x, y) = E(x,y)∼pl (x,y) [− log qφ (y|x)](5)where pl (x, y) is the distribution of labeled data.
To sum up, the semi-supervised model (Semisup) is trained to maximize the variational lower
bounds and minimize the classification crossentropy error of 5.

✏ ⇠ N (0, 1)

Source Word⌃(x)

Reinflected Form......

z

µ(x)

k

<w>

+
k

a

b

l

y1T

L(x(s) , x(t) , y(t) , x) = α · U(x) + Lu (x(s) |x(t) )+ Ll (x(t) , y(t) |x(s) ) − D(x(t) , y(t) )

ä

k

y2T

y3T

y4T ....

Supervised Variational Encoder Decoder(6)✏ ⇠ N (0, 1) k

Source Word⌃(x)

The weight α controls the relative weight between
the loss from unlabeled data and labeled data.

µ(x)aSource Word......

zk<w>
Multinomial Sampling3

y1

Learning MSVED

k

a

l

b

y2

Learning Discrete Latent Variablesy3

One challenge in training our model is that discrete random variables in a stochastic computation graph prevent the gradient from being backpropagated due to their non-differentiability, andmarginalizing over all label combinations is also
infeasible in our case.
To alleviate this problem, we use the recently
proposed Gumbel-Softmax trick (Maddison et al.,2014; Gumbel and Lieblein, 1954) to create a differentiable estimator for categorical variables. In
experiments, we start with a relatively large temperature and decrease it gradually.

···

3.1

3.2

Learning Continuous Latent Variables

We observe that with the vanilla implementation
the KL cost quickly decreases to near zero, setting
qφ (z|x) equal to standard normal distribution. In
this case, the RNN decoder can easily degenerate
into an RNN language model. Hence, the latentvariables are ignored by the decoder and cannot
encode any useful information. The latent variable z learns an undesirable distribution that coincides with the imposed prior distribution but has
no contribution to the decoder. To force the decoder to use the latent variables, we take the following two approaches which are similar to Bowman et al. (2016).
KL-Divergence Annealing: We add a coefficient
λ to the KL cost and gradually anneal it from zero
to a predefined threshold λm . At the early stageof training, we set λ to be zero and let the model
first figure out how to project the representation of
the source sequence to a roughly right point in thespace and then regularize it with the KL cost. This
technique can also be seen in (Kočiskỳ et al., 2016;
Miao and Blunsom, 2016).
Input Dropout in the Decoder: Besides annealing the KL cost, we also randomly drop out they1 2 {pos: V, N, ADJ}..

y4

y2 2 {def: DEF, INDEF}+

y3 2 {num: DU, SG, PL}...
...
...

Unsupervised Variational Auto-encoder

Figure 1: Model architecture for labeled and unlabeled data. For the encoder-decoder model, only
one direction from the source to target is given.
The classification model is not illustrated in the
diagram.
input token with a probability of β at each time
step of the decoder during learning. The previous
ground-truth token embedding is replaced with azero vector when dropped. In this way, the RNN
decoder could not fully rely on the ground-truth
previous token, which ensures that the decoder
uses information encoded in the latent variables.

4

Architecture for Morphological
Reinflection

The overall model architecture is shown in Fig. 1.
Each character and each label is associated witha continuous vector. We employ Gated Recurrent Units (GRUs) for the encoder and decoder.
We use only single directional GRUs as the encoder for the input word x(s) . u is the hidden representation of x(s) which is the last hidden stateof GRUs. and is used as the input for the inference model on z. We represent µ(u) and σ 2 (u) as
MLPs and sample z from N (µ(u), diag(σ 2 (u))),using z = µ + σ ◦ , where  ∼ N (0, I). Similarly, we can obtain the hidden representation ofx(t) and use this as input to the inference model on(t)each label yi , which is also an MLP following asoftmax layer to generate the categorical probabilities of target labels.
Other experimental setups: We apply temperature annealing in the Gumble-Softmax with thescheme max(0.5, exp(−3e − 5 · t)) every 2000
updates where t is the update steps. We observe

Language
Latin
Icelandic
Irish
FinnishHungarianFaroeseRussianNorwegian-Nynorsk
PolishGermanSwedishRomanian
Lithuanian
Serbo-Croatian
Norwegian-Bokmal
Czech
KurmanjiUkrainian

Dev
66.2
71.7
72.7
73.4
74.5
74.8
75.8
77.8
78.3
79.3
80.2
80.3
80.6
81.1
81.2
83.1
83.4
84.5

Test
66.2
68.1
71.9
74.9
73.6
74.5
76.4
73.8
78.1
78.7
80.6
78.6
81.6
79.6
82
81.9
83.8
84.0

Language
NavajoFrenchArmenian
LatvianScottish-Gaelic
BulgarianMacedonian
Northern-Sami
SloveneDanishArabicSorani
SlovakTurkishDutch
AlbanianGeorgian
Spanish

Dev
84.9
84.9
85.3
85.6
86.0
86.3
86.6
86.7
87.3
88.1
88.6
89.6
89.6
90.4
91.2
91.9
92.5
92.5

Test
84.2
82.4
82.3
87.5
68.0
86.7
86.1
85.8
87.8
85.4
85.9
87.8
87.9
90.3
88.9
91.3
92.3
92.8

Language
EnglishLower-Sorbian
ItalianBasqueEstonian
Quechua
Khaling
Hebrew
Portuguese
Catalan
Urdu
PersianBengali
Welsh
Haida
Hindi

Dev
93.3
93.9
94.2
95.0
95.1
95.5
96.2
96.3
96.4
96.9
98.4
98.6
99.0
99.0
99.0
99.9

Test
94.6
91.3
92.6
97.0
93.7
95.5
94.8
97.5
96.4
96.5
97.9
98.7
99.0
99.0
97.0
99.6

Average

87.18

86.21

Table 1: Results of the ensemble system on the development ang test sets of 52 languages.
Language

Src Word

Tgt Labels

Latintrygonlargiocompenso

Pos=N;Case=ABL;Num=PL
Mood=SBJV;Num=PL;Per=2;Tense=PST;Asp=PRF;Pos=V
Mood=SBJV;Num=SG;Per=3;Tense=PST;Asp=PFV;Pos=V

Icelandic

háspilgallabuxur
lest

Pos=N;Def=DEF;Case=GEN;Num=PL
Pos=N;Def=INDF;Case=GEN;Num=SG
Pos=N;Def=DEF;Case=GEN;Num=SG L

Gold Tgt

Ourstrȳgōnibus
largīvissẽtiscompensāverit

trygõnibus
largissētiscompenserit

háspilanna
gallabuxna
lestarinnarhásplanna
gallabölurlestsins

Table 2: Examples of incorrect inflection generation words on the dev data.
Language

Settings

Dev Acc. (Single Model.)Icelandic

vanilla Encoder-Decoder + attention, w/o data augmentation
our model w/o data augmentation and Wiki
our model (full)

81.0
78.6
71.7

Latin

vanilla Encoder-Decoder + attention, w/o data augmentation
our model w/o data augmentation and Wiki
our model (full)

74.6
66.6
66.2

Persianvanilla Encoder-Decoder + attention, w/o data augmentation
our model w/o data augmentation and Wiki
our model (full)

99.6
99.6
98.6

Arabicvanilla Encoder-Decoder + attention, w/o data augmentation
our model w/o data augmentation and Wiki
our model (full)

90.7
91.3
88.6

Table 3: Ablation experiments on the effects of data augmentation and WikiData.
that our model is not sensitive to the temperaturein this task. All hyperparameters are tuned on thevalidation set, and include the following: For KL
cost annealing, λm is set to be 0.2 for all language
settings. For character drop-out at the decoder, we

empirically set β to be 0.4 for all languages. We
set the dimension of character embeddings to be
300, tag label embeddings to be 200, RNN hidden state to be 256, and latent variable z to be 150
or 100. We set α the weight for the unsupervised

loss to be 0.8. We train the model with Adadelta(Zeiler, 2012) and use early-stop with a patienceof 5. Our system is an ensemble of five models
and the probability vector at each time step is obtained by averaging the output probabilities fromeach model

5

Experiments

5.1

Data pre-processing

Creating morphosyntactic tag maps: In ourmodel, we treat the inference model on discrete labels in the form of discriminator, thus we need toknow which label belongs to which morphosyntactic dimension. For example, V is a label ofPart-of-speech-tagging. To obtain such mappingfrom a specific label to the morphosyntactic dimension, we leverage the Universal Morphological Feature Schema (Sylak-Glassman, 2016) andalso add the missing schema from the training data
to create the key-value pairs of morphosysntactic
dimension and label. Then we reformat the labels
provided in the data set into the key-value pairs totrain a classifier for each morphosyntactic dimension.
Data Augmentation: We augment the data set in
the similar way as Kann and Schütze (2016). By
doing so, the training data is not limited to theform of lemma to inflected word but can also be
any word pairs that share the same lemma. This
helps our model generalize better and learn the latent continuous representations more effectively.
The size of training data set after augmentation
scales with a factor of 2 to 20 times compared withthe original one.
Monolingual WikiData:
We process the
Wikipedia corpus provided by the shared task organizer as our unsupervised training data together
with words in the training data. For each language,
we first get the character vocabulary of the corresponding training data and only keep words inthe Wiki corpus for which characters are all in thecharacter set we obtained. All words that occur
less than 20 times are eliminated. We also limit
the number of words used during training to be the50000 most frequent words.
5.2

Results and Analysis

The results on the dev and test data of the 52 languages are presented in 1. We obtain a generation accuracy above 80% over more than 25%
languages and an average of 87.2% for both dev

and test data. The generation accuracy is almost
consistent on the dev and test data except that thetest data accuracy of Scottish-Gaelic drops by near
21%. We find that only a medium volume of training data is provided for Scottish-Gaelic. This may
be the reason why the model trained for ScottishGaelic can not generalize as well as other languages.
We do not tune the hyper-parameters for eachlanguage manually. However, we test on different dimensions for the continuous latent variables. The dimension size we have used included
100 and 150. And we observe significant improvement by using a larger dimension size of latent variables over a portion of languages including Faroese, Lithuanian, Navajo, Scottish-gaelic,
Northern-sami, Slovene, Sorani, Slovak. However, we also observe that for some languages including Finnish, German, French, etc, the performance drops signficantly after increasing the sizeof continuous latent variable dimension. This indicates that for different languages, the continuous
space required to encode the lemma and inflected
information varies from language to language. We
will further investigate this in the future work.
5.3

Effect of Data Augmentation and Using
Wiki Data

While our performance was reasonable, it was not
as good as that presented in our previous work
(Zhou and Neubig, 2017), nor was it competitive with the highest-scoring models on the sharedtask. In order to examine the reason for this, we
performed several ablations, the results of which
are presented in Tab. 3First, we first examined the effects of data
augmentation and Wiki Data for semi-supervised
learning on the performance of our model. By removing the augmented data from the training set,
we observe a large gain in the generation accuracy. Besides, we find that Wiki Data for semisupervised learning doesn’t help much to increase
the model’s performance. The reasons for this will
be examined further in the following section.
We additionally reimplemented a vanilla
encoder-decoder model with attention that concatenates the input characters and target wordtags together with a special token in the middle
as the new input sequence to the encoder (Kannand Schütze, 2016). The results show that thevanilla encoder-decoder works better than ourDimension

Case
Possession
Language-Specific-Features
Mood

Definiteness
GenderPoliteness
Number

PersonTense
AspectPart-of-Speech

Voice

Label
None
ACC
NOM
GEN
None
PSSD
None
LGSPEC1
None
IND
IMP
SBJV
None
DEF
NDEF
None
FEM
MASC
None
INFM
None
DU
SG
PL
None
1
3
2
None
PST
None
PRF
IPFV
None
V+V.PTCP
V+V.MSDR
N
ADJ
V
None
PASS
ACT

Train Data

WikiData

Difference

0.58
0.14
0.14
0.14
0.86
0.14
0.90
0.10
0.68
0.20
0.02
0.10
0.57
0.22
0.21
0.53
0.23
0.23
0.85
0.14
0.01
0.22
0.47
0.30
0.58
0.06
0.18
0.17
0.90
0.10
0.80
0.10
0.10
0.00
0.01
0.00
0.43
0.14
0.42
0.57
0.16
0.27

0.35
0.51
0.12
0.01
0.31
0.69
0.42
0.58
0.10
0.62
0.03
0.25
0.60
0.34
0.06
0.52
0.27
0.20
0.58
0.42
0.16
0.34
0.31
0.18
0.74
0.02
0.17
0.08
0.51
0.49
0.21
0.41
0.38
0.03
0.29
0.15
0.36
0.14
0.03
0.40
0.39
0.22

-0.22
0.38
-0.02
-0.13
-0.55
0.55
-0.48
0.48
-0.58
0.42
0.01
0.15
0.03
0.12
-0.15
-0.01
0.04
-0.03
-0.28
0.28
0.15
0.12
-0.15
-0.11
0.15
-0.05
-0.01
-0.09
-0.40
0.40
-0.59
0.31
0.28
0.03
0.28
0.14
-0.07
-0.00
-0.39
-0.18
0.22
-0.05

Table 4: The distribution of morphosyntactic tags for Arabic on Wikipedia and the shared task training
data respectively. The linguistic tag classifier has an average accuracy of 93.36% on the Dev data.
model in some cases. We suspect that sincetask 1 is purely an inflection task and because
semi-supervised learning did not provide a particularly large benefit, a simpler model that utilizes
attention may be sufficient. This is in contrast
to our previous findings, where semi-supervised
learning was highly effective, and the proposedmodel out-performed the simpler attention-based
baseline.
5.4

Analysis on the Distribution of Linguistic
Tags of Wiki Data and Training Data

One potential reason for the lack of effectiveness of semi-supervised training is that the semi-supervised data that we used for training was not
appropriate for the task at hand, or that we were
not able to use it in the most effective way. In
order to do so, we analyze the distribution of linguistic tags for words from the training data in theshared task and the Wiki Data provided by the organizer, with the hypothesis that if the distributionof tags for the Wiki Data is very different from thetraining and test data for the shared task, our predictions may be biased away from the testing distribution by incorporating the unsupervised Wikidata. To perform this examination, we use the tagclassifier trained in our model to predict the labels
for each word in the Wiki Data.

Dimension
Mood
Politeness
Number

PersonFiniteness
Tense

AspectPart-of-Speech

Label

Train Data

WikiData

Difference

0.79
0.03
0.18
0.52
0.48
0.04
0.48
0.47
0.04
0.31
0.31
0.34
0.98
0.02
0.13
0.04
0.46
0.36
0.39
0.18
0.17
0.18
0.09
0.00
0.03
0.97

0.13
0.69
0.18
0.30
0.70
0.67
0.19
0.15
0.28
0.23
0.22
0.27
0.33
0.67
0.07
0.42
0.14
0.37
0.37
0.07
0.03
0.09
0.44
0.44
0.18
0.38

-0.66
0.66
-0.00
-0.22
0.22
0.62
-0.30
-0.32
0.24
-0.08
-0.09
-0.07
-0.66
0.66
-0.07
0.38
-0.32
0.01
-0.01
-0.11
-0.14
-0.08
0.35
0.44
0.15
-0.59

None
IMP
SBJV
None
COL
None
SG
PL
None
1
3
2
None
NFIN
None
FUT
PST
PRS
None
PROG
PRF
IPFV
PFV
None
V+V.PTCP
V

Table 5: The distribution of morphosyntactic tags for Persian on Wikipedia and the shared task training
data respectively. The linguistic tag classifier has an average accuracy of 95.26% on the Dev data.
The percentages of each label within each morphosyntactic dimension for Arabic and Persian arelisted in Tab. 4 and Tab. 5. We found thatthe distribution of the linguistic tags for the Wiki
Data and the training data in the shared task are
not always consistent. For example, in Arabic,
the distributions of predicted tags with respect tocase, possession, part-of-speech, and several otherclasses differ significantly from the original training data. Such difference suggests that either thewords in the unlabeled Wiki Data have very different characteristics than our training set, or ourtag classifier is not functioning properly to identify the tags. Either case would be detrimental
to semi-supervised learning. The problem is evenmore stark for Persian: in Persian the only labeled
words in the training data are verbs, so all nonverb words in the Wiki Data will receive an incorrect analysis, which is obviously not conducive tolearning anything useful. As a recommendation
for the future, when performing semi-supervised
learning for morphology where the labeled data
only represents a subset of the phenomena in thelanguage, it is likely necessary to first identify
which of the available unlabeled data is appropriate for semi-supervised learning before applying

such methods.
5.5

Case Study on Inflected Words

In Tab. 1, we notice that the performance on
Latin is relatively poor compared with other languages. Latin is a highly inflected languages withthree distinct genders, seven noun cases, four verb
conjugations, four verb principal parts, six tenses,
three persons, three moods, two voices, two aspects and two numbers. In addition to this, wefound that the data set size after augmentation was
only enlarged 2 times. We examine some errors
made by our system on two worst performed languages Latin and Icelandic in Tab. 2. As shownin the table, we found that the inflections of Latin
and Icelandic have more suffix variations from thelemma. We guess our model still lacks the ability
to capture more complicated inflections for suchlanguages. We might consider adding the dependencies between different inflections for multiple
target labels in our future work.

6

Conclusion and Future WorkIn this work, we further examine the method proposed in (Zhou and Neubig, 2017) for the sharedtask of SIGMORPHON 2017 on 52 languages anddemonstrate the effectiveness of this approach. We
will further improve our model’s sophistication by
investigating strategies for choosing appropriate
semi-supervised data, and examining the model’s
performance on languages with a high inflectionlevel.

Tomáš Kočiskỳ, Gábor Melis, Edward Grefenstette,
Chris Dyer, Wang Ling, Phil Blunsom, andKarl Moritz Hermann. 2016. Semantic parsing withsemi-supervised sequential autoencoders. the 2016
Conference on Empirical Methods in Natural Language Processing (EMNLP) .

Acknowledgments

Chris J Maddison, Daniel Tarlow, and Tom Minka.
2014. A* sampling. In Advances in Neural Information Processing Systems. pages 3086–3094.

This work has been supported in part by an
Amazon Academic Research Award. We thankMatthew Honnibal for pointing out that the data
distribution of Wikipedia corpus might be biased.

Yishu Miao and Phil Blunsom. 2016. Language as alatent variable: Discrete generative models for sentence compression. the 2016 Conference on Empirical Methods in Natural Language Processing(EMNLP) .

References

John Sylak-Glassman. 2016. The composition and useof the universal morphological feature schema (unimorph schema) .

Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous
space. Proceedings of CoNLL .

Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 .

Victor Chahuneau, Eva Schlinger, Noah A Smith, andChris Dyer. 2013. Translating into morphologicallyrich languages with synthetic phrases. Association
for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman,David Yarowsky, Jason Eisner, and Mans Hulden.
2016.
The sigmorphon 2016 shared task—
morphological reinflection. In Proceedings of the2016 Meeting of SIGMORPHON. Association for
Computational Linguistics, Berlin, Germany.
Ryan Cotterell and Hinrich Schütze. 2017. Joint semantic synthesis and morphological analysis of thederived word. arXiv preprint arXiv:1701.00946 .
Kareem Darwish and Douglas W Oard. 2007. Adapting morphology for arabic information retrieval. In
Arabic Computational Morphology, Springer, pages
245–262.
Carl Doersch. 2016. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908 .
Emil Julius Gumbel and Julius Lieblein. 1954. Statistical theory of extreme values and some practicalapplications: a series of lectures. US Government
Printing Office Washington .
Katharina Kann and Hinrich Schütze. 2016. Med: The
lmu system for the sigmorphon 2016 shared task onmorphological reinflection. In In Proceedings of the
14th SIGMORPHON Workshop on Computational
Research in Phonetics, Phonology, and Morphology. Berlin, Germany.
D.P. Kingma and M. Welling. 2014. Auto-encoding
variational bayes. In The International Conference
on Learning Representations.

Chunting Zhou and Graham Neubig. 2017. Multispace variational encoder-decoders for semisupervised labeled sequence transduction. In The
55th Annual Meeting of the Association for Computational Linguistics (ACL). Vancouver, Canada.
https://arxiv.org/abs/1704.01691.