Venue: ACL 2017

# Multi-space Variational Encoder-Decoders

for Semi-supervised Labeled Sequence Transduction

Chunting Zhou, Graham Neubig

Language Technologies Institute

Carnegie Mellon University

ctzhou,gueubig@cs.cmu.edu

###### Abstract

Labeled sequence transduction is a task of transforming one sequence into another sequence that satisfies desiderata specified by a set of labels. In this paper we propose _multi-space variational encoder-decoders_, a new model for labeled sequence transduction with semi-supervised learning. The generative model can use neural networks to handle both discrete and continuous latent variables to exploit various features of data. Experiments show that our model provides not only a powerful supervised framework but also can effectively take advantage of the unlabeled data. On the SIGMORPHON morphological inflection benchmark, our model outperforms single-model state-of-art results by a large margin for the majority of languages.1

Footnote 1: An implementation of our model are available at [https://github.com/violet-zct/MSVED-morph-reinflection](https://github.com/violet-zct/MSVED-morph-reinflection).

+
Footnote †: _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_, pages 310–320 Vancouver, Canada, July 30 - August 4, 2017. ©2017 Association for Computational Linguistics

## 1 Introduction

This paper proposes a model for labeled sequence transduction tasks, tasks where we are given an input sequence and a set of labels, from which we are expected to generate an output sequence that reflects the content of the input sequence and desiderata specified by the labels. Several examples of these tasks exist in prior work: using labels to moderate politeness in machine translation results (Sennrich et al., 2016), modifying the output language of a machine translation system (Johnson et al., 2016), or controlling the length of a summary in summarization (Kikuchi et al., 2016). In particular, however, we are motivated by the task of morphological reinflection (Cotterell et al., 2016), which we will use as an example in our description and test bed for our models.

In morphologically rich languages, different affixes (i.e. prefixes, infixes, suffixes) can be combined with the lemma to reflect various syntactic and semantic features of a word. The ability to accurately analyze and generate morphological forms is crucial to creating applications such as machine translation (Chahuneau et al., 2013; Toutanova et al., 2008) or information retrieval (Darwish and Oard, 2007) in these languages. As shown in 1, \({\color[rgb]{0,0,0}r}\)-infection of an inflected form given the target linguistic labels is a challenging sub-task of handling morphology as a whole, in which we take as input an inflected form (in the example, "playing") and labels representing the desired form ("pos=Verb, tense=Past") and must generate the desired form ("played").

Approaches to this task include those utilizing hand-crafted linguistic rules and heuristics (Taji et al., 2016), as well as learning-based approaches using alignment and extracted transduction rules (Durrett and DeNero, 2013; Alegria and Ekceberria, 2016; Nicolai et al., 2016). There have also been methods proposed using neural sequence-to-sequence models (Faruqui et al., 2016; Kann et al., 2016; Ostling, 2016), and currently ensembles of attentional encoder-decoder models (Kann and Schutze, 2016a, b) have achieved state-of-art results on this task. One feature of these neural models however, is that they are trained in a

Figure 1: Standard supervised labeled sequence transduction, and our proposed semi-supervised method.



largely _supervised_ fashion (top of Fig. 1), using data explicitly labeled with the input sequence and labels, along with the output representation. Needless to say, the ability to obtain this annotated data for many languages is limited. However, we can expect that for most languages we can obtain large amounts of unlabeled surface forms that may allow for _semi-supervised_ learning over this unlabeled data (entirety of Fig. 1).2

Footnote 2: Faruqui et al. (2016) have attempted a limited form of semi-supervised learning by re-ranking with a standard \(n\)-gram language model, but this is not integrated with the learning process for the neural model and gains are limited.

In this work, we propose a new framework for labeled sequence transduction problems: multi-space variational encoder-decoders (**MSVED**, SS3.3). MSVEDs employ continuous or discrete latent variables belonging to multiple separate probability distributions3 to explain the observed data. In the example of morphological reinflection, we introduce a vector of continuous random variables that represent the lemma of the source and target words, and also one discrete random variable for each of the labels, which are on the source or the target side.

Footnote 3: Analogous to multi-space hidden Markov models Tokuda et al. (2002)

This model has the advantage of both providing a powerful modeling framework for supervised learning, and allowing for learning in an unsupervised setting. For labeled data, we maximize the variational lower bound on the marginal log likelihood of the data and annotated labels. For unlabeled data, we train an auto-encoder to reconstruct a word conditioned on its lemma and morphological labels. While these labels are unavailable, a set of discrete latent variables are associated with each unlabeled word. Afterwards we can perform posterior inference on these latent variables and maximize the variational lower bound on the marginal log likelihood of data.

Experiments on the SIGMORPHON morphological reinflection task Cotterell et al. (2016) find that our model beats the state-of-the-art for a single model in the majority of languages, and is particularly effective in languages with more complicated inflectional phenomena. Further, we find that semi-supervised learning allows for significant further gains. Finally, qualitative evaluation of lemma representations finds that our model is able to learn lemma embeddings that match with human intuition.

## 2 Labeled Sequence Transduction

In this section, we first present some notations regarding labeled sequence transduction problems in general, then describe a particular instantiation for morphological reinflection.

Notation:Labeled sequence transduction problems involve transforming a source sequence \({\bf x}^{(s)}\) into a target sequence \({\bf x}^{(t)}\), with some labels describing the particular variety of transformation to be performed. We use discrete variables \(y_{1}^{(t)},y_{2}^{(t)},\cdots,y_{K}^{(t)}\) to denote the labels associated with each target sequence, where \(K\) is the total number of labels. Let \({\bf y}^{(t)}=[y_{1}^{(t)},y_{2}^{(t)},\cdots,y_{K}^{(t)}]\) denote a vector of these discrete variables. Each discrete variable \(y_{k}^{(t)}\) represents a categorical feature pertaining to the target sequence, and has a set of possible labels. In the later sections, we also use \({\bf y}^{(t)}\) and \(y_{k}^{(t)}\) to denote discrete latent variables corresponding to these labels.

Given a source sequence \({\bf x}^{(s)}\) and a set of associated target labels \({\bf y}^{(t)}\), our goal is to generate a target sequence \({\bf x}^{(t)}\) that exhibits the features specified by \({\bf y}^{(t)}\) using a probabilistic model \(p({\bf x}^{(t)}|{\bf x}^{(s)},{\bf y}^{(t)})\). The best target sequence \(\hat{\bf x}^{(t)}\) is then given by:

\[\hat{\bf x}^{(t)}=\operatorname*{arg\,max}_{{\bf x}^{(t)}}p({\bf x}^{(t)}|{ \bf x}^{(s)},{\bf y}^{(t)}). \tag{1}\]

Morphological Reinflection Problem:In morphological reinflection, the source sequence \({\bf x}^{(s)}\) consists of the characters in an inflected word (e.g., "played"), while the associated labels \({\bf y}^{(t)}\) describe some linguistic features (e.g., \(y_{\text{pos}}^{(t)}=\texttt{Verb}\), \(y_{\text{sense}}^{(t)}=\texttt{Past}\)) that we hope to realize in the target. The target sequence \({\bf x}^{(t)}\) is therefore the characters of the re-inflected form of the source word (e.g., "played") that satisfy the linguistic features specified by \({\bf y}^{(t)}\). For this task, each discrete variable \(y_{k}^{(t)}\) has a set of possible labels (e.g. pos=V, pos=ADJ, etc) and follows a multinomial distribution.

## 3 Proposed Method

### Preliminaries: Variational Autoencoder

As mentioned above, our proposed model uses probabilistic latent variables in a model based on neural networks. The variational autoencoder Kingma and Welling (2014) is an efficient way to handle (continuous) latent variables in neural 

models. We describe it briefly here, and interested readers can refer to Doersch (2016) for details. The VAE learns a generative model of the probability \(p({\bf x}|{\bf z})\) of observed data \({\bf x}\) given a latent variable \({\bf z}\), and simultaneously uses a recognition model \(q({\bf z}|{\bf x})\) at learning time to estimate \({\bf z}\) for a particular observation \({\bf x}\) (Fig. 2(a)). \(q(\cdot)\) and \(p(\cdot)\) are modeled using neural networks parameterized by \(\phi\) and \(\theta\) respectively, and these parameters are learned by maximizing the variational lower bound on the marginal log likelihood of data:

\[\log p_{\theta}({\bf x}) \geq\mathbb{E}_{{\bf z}\sim q_{\phi}({\bf z}|{\bf x})}[\log p_{ \theta}({\bf x}|{\bf z})]-\] \[\text{KL}(q_{\phi}({\bf z}|{\bf x})||p({\bf z})) \tag{2}\]

The KL-divergence term (a standard feature of variational methods) ensures that the distributions estimated by the recognition model \(q_{\phi}({\bf z}|{\bf x})\) do not deviate far from our prior probability \(p({\bf z})\) of the values of the latent variables. To optimize the parameters with gradient descent, Kingma and Welling (2014) introduce a reparameterization trick that allows for training using simple backpropagation w.r.t. the Gaussian latent variables \({\bf z}\). Specifically, we can express \({\bf z}\) as a deterministic variable \({\bf z}=g_{\phi}(\epsilon,{\bf x})\) where \(\epsilon\) is an independent Gaussian noise variable \(\epsilon\sim\mathcal{N}(0,1)\). The mean \(\mu\) and the variance \(\sigma^{2}\) of \({\bf z}\) are reparameterized by the differentiable functions w.r.t. \(\phi\). Thus, instead of generating \({\bf z}\) from \(q_{\phi}({\bf z}|{\bf x})\), we sample the auxiliary variable \(\epsilon\) and obtain \({\bf z}=\mu_{\phi}(x)+\sigma_{\phi}(x)\circ\epsilon\), which enables gradients to backpropagation through \(\phi\).

### Multi-space Variational Autoencoders

As an intermediate step to our full model, we next describe a generative model for a single sequence with both continuous and discrete latent variables, the multi-space variational auto-encoder (MSVAE). MSVAEs are a combination of two threads of previous work: deep generative models with both continuous/discrete latent variables for classification problems (Kingma et al., 2014; Maaloe et al., 2016) and VAEs with only continuous variables for sequential data (Bowman et al., 2016; Chung et al., 2015; Zhang et al., 2016; Fabius and van Amersfoort, 2014; Bayer and Osendorfer, 2014). In MSVAEs, we have an observed sequence \({\bf x}\), continuous latent variables \({\bf z}\) like the VAE, as well as discrete variables \({\bf y}\).

In the case of the morphology example, \({\bf x}\) can be interpreted as an inflected word to be generated. \({\bf y}\) is a vector representing its linguistic labels, either annotated by an annotator in the observed case, or unannotated in the unobserved case. \({\bf z}\) is a vector of latent continuous variables, e.g. a latent embedding of the lemma that captures all the information about \({\bf x}\) that is not already represented in labels \({\bf y}\).

**MSVAE**: Because inflected words can be naturally thought of as "lemma+morphological labels", to interpret a word, we resort to discrete and continuous latent variables that represent the linguistic labels and the lemma respectively. In this case when the labels of the sequence \({\bf y}\) is not observed, we perform inference over possible linguistic labels and these inferred labels are referenced in generating \({\bf x}\).

The generative model \(p_{\theta}({\bf x},{\bf y},{\bf z})=p({\bf z})p_{\pi}({\bf y})p_{\theta}({\bf x }|{\bf y},{\bf z})\) is defined as:

\[p({\bf z}) =\mathcal{N}({\bf z}|{\bf 0},{\bf I}) \tag{3}\] \[p_{\pi}({\bf y}) =\prod_{k}\text{Cat}(y_{k}|\pi_{k})\] (4) \[p_{\theta}({\bf x}|{\bf y},{\bf z}) =f({\bf x};{\bf y},{\bf z},\theta). \tag{5}\]

Like the standard VAE, we assume the prior of the latent variable \({\bf z}\) is a diagonal Gaussian distribution with zero mean and unit variance. We assume that each variable in \({\bf y}\) is independent, resulting in a factorized distribution in Eq. 4, where \(\text{Cat}(y_{k}|\pi_{k})\) is a multinomial distribution with parameters \(\pi_{k}\). For the purposes of this study, we set these to a uniform distribution \(\pi_{k,j}=\frac{1}{|\pi_{k}|}\). \(f({\bf x};{\bf y},{\bf z},\theta)\) calculates the likelihood of \({\bf x}\), a function parametrized by deep neural networks. Specifically, we employ an RNN decoder to generate the target word conditioned on the lemma variable \({\bf z}\) and linguistic labels \({\bf y}\), detailed in SS5.

When inferring the latent variables from the given data \({\bf x}\), we assume the joint distribution of latent variables \({\bf z}\) and \({\bf y}\) has a factorized form, i.e. \(q({\bf z},{\bf y}|{\bf x})=q({\bf z}|{\bf x})q({\bf y}|{\bf x})\) as shown in Fig. 2(c).

Figure 2: Graphical models of (a) VAE, (b) labeled MSVAE, (c) MSVAE, (d) labeled MSVED, and (e) MSVDER. White circles are latent variables and shaded circles are observed variables. Dashed lines indicate the inference process while the solid lines indicate the generative process.

 

The inference model is defined as follows:

\[q_{\phi}(\mathbf{z}|\mathbf{x}) =\mathcal{N}(\mathbf{z}|\mu_{\phi}(\mathbf{x}),\text{diag}(\sigma^{2 }_{\phi}(\mathbf{x}))) \tag{6}\] \[q_{\phi}(\mathbf{y}|\mathbf{x}) =\prod_{k}q_{\phi}(y_{k}|\mathbf{x})\] \[=\prod_{k}\text{Cat}(y_{k}|\pi_{\phi}(\mathbf{x})) \tag{7}\]

where the inference distribution over \(\mathbf{z}\) is a diagonal Gaussian distribution with mean and variance parameterized by neural networks. The inference model \(q(\mathbf{y}|\mathbf{x})\) on labels \(\mathbf{y}\) has the form of a discriminative classifier that generates a set of multinomial probability vectors \(\pi_{\phi}(\mathbf{x})\) over all labels for each tag \(y_{k}\). We represent each multinomial distribution \(q(y_{k}|\mathbf{x})\) with an MLP.

The MSVAE is trained by maximizing the following variational lower bound \(\mathcal{U}(\mathbf{x})\) on the objective for unlabeled data:

\[\log p_{\theta}(\mathbf{x}) \geq\mathbb{E}_{(\mathbf{y},\mathbf{z})\sim q_{\phi}(\mathbf{y}, \mathbf{z}|\mathbf{x})}\log\frac{p_{\theta}(\mathbf{x},\mathbf{y},\mathbf{z}) }{q_{\phi}(\mathbf{y},\mathbf{z}|\mathbf{x})}\] \[=\mathbb{E}_{\mathbf{y}\sim q_{\phi}(\mathbf{y}|\mathbf{x})}[ \mathbb{E}_{\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x})}[\log p_{\theta}( \mathbf{x}|\mathbf{z},\mathbf{y})]\] \[-\text{KL}(q_{\phi}(\mathbf{z}|\mathbf{x})||p(\mathbf{z}))+\log p _{\pi}(\mathbf{y})\] \[-\log q_{\phi}(\mathbf{y}|\mathbf{x})]=\mathcal{U}(\mathbf{x}) \tag{8}\]

Note that this introduction of discrete variables requires more sophisticated optimization algorithms, which we will discuss in SS4.1.

**Labeled MSVAE**: When \(\mathbf{y}\) is observed as shown in Fig. 2(b), we maximize the following variational lower bound on the marginal log likelihood of the data and the labels:

\[\log p_{\theta}(\mathbf{x},\mathbf{y}) \geq\mathbb{E}_{\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x})} \log\frac{p_{\theta}(\mathbf{x},\mathbf{y},\mathbf{z})}{q_{\phi}(\mathbf{z}| \mathbf{x})}=\] \[\mathbb{E}_{\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x})}[\log p _{\theta}(\mathbf{x}|\mathbf{y},\mathbf{z})+\log p_{\pi}(\mathbf{y})]\] \[~{}~{}~{}~{}~{}-\text{KL}(q_{\phi}(\mathbf{z}|\mathbf{x})||p( \mathbf{z})) \tag{9}\]

which is a simple extension to Eq. 2.

Note that when labels are not observed, the inference model \(q_{\phi}(\mathbf{y}|\mathbf{x})\) has the form of a discriminative classifier, thus we can use observed labels as the supervision signal to learn a better classifier. In this case we also minimize the following cross entropy as the classification loss:

\[\mathcal{D}(\mathbf{x},\mathbf{y})=\mathbb{E}_{(\mathbf{x},\mathbf{y})\sim p _{l}(\mathbf{x},\mathbf{y})}[-\log q_{\phi}(\mathbf{y}|\mathbf{x})] \tag{10}\]

where \(p_{l}(\mathbf{x},\mathbf{y})\) is the distribution of labeled data. This is a form of multi-task learning, as this additional loss also informs the learning of our representations.

### Multi-space Variational Encoder-Decoders

Finally, we discuss the full proposed method: the multi-space variational encoder-decoder (MSVED), which generates the target \(\mathbf{x}^{(t)}\) from the source \(\mathbf{x}^{(s)}\) and labels \(\mathbf{y}^{(t)}\). Again, we discuss two cases of this model: labels of the target sequence are observed and not observed.

**MSVED:** The graphical model for the MSVED is given in Fig. 2 (e). Because the labels of target sequence are not observed, once again we treat them as discrete latent variables and make inference on the these labels conditioned on the target sequence. The generative process for the MSVD is very similar to that of the MSVAE with one important exception: while the standard MSVAE conditions the recognition model \(q(\mathbf{z}|\mathbf{x})\) on \(\mathbf{x}\), then generates \(\mathbf{x}\) itself, the MSVED conditions the recognition model \(q(\mathbf{z}|\mathbf{x}^{(s)})\) on the source \(\mathbf{x}^{(s)}\), then generates the target \(\mathbf{x}^{(t)}\). Because only the recognition model is changed, the generative equations for \(p_{\theta}(\mathbf{x}^{(t)},\mathbf{y}^{(t)},\mathbf{z})\) are exactly the same as Eqs. 3-5 with \(\mathbf{x}^{(t)}\) swapped for \(\mathbf{x}\) and \(\mathbf{y}^{(t)}\) swapped for \(\mathbf{y}\). The variational lower bound on the conditional log likelihood, however, is affected by the recognition model, and thus is computed as:

\[\log p_{\theta}(\mathbf{x}^{(t)}|\mathbf{x}^{(s)})\] \[\geq \mathbb{E}_{(\mathbf{y}^{(t)},\mathbf{z})\sim q_{\phi}(\mathbf{y} ^{(t)},\mathbf{z}|\mathbf{x}^{(s)},\mathbf{x}^{(t)})}\log\frac{p_{\theta}( \mathbf{x}^{(t)},\mathbf{y}^{(t)},\mathbf{z}|\mathbf{x}^{(s)})}{q_{\phi}( \mathbf{y}^{(t)},\mathbf{z}|\mathbf{x}^{(s)},\mathbf{x}^{(t)})}\] \[= \mathbb{E}_{\mathbf{y}^{(t)}\sim q_{\phi}(\mathbf{y}^{(t)}|\mathbf{ x}^{(t)})}[\mathbb{E}_{\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x}^{(s)})}[\log p_{ \theta}(\mathbf{x}^{(t)}|\mathbf{y}^{(t)},\mathbf{z})]\] \[-\text{KL}(q_{\phi}(\mathbf{z}|\mathbf{x}^{(s)})||p(\mathbf{z})) \pm\log p_{\pi}(\mathbf{y}^{(t)})\] \[-\log q_{\phi}(\mathbf{y}^{(t)}|\mathbf{x}^{(t)})]=\mathcal{L}_{u} (\mathbf{x}^{(t)}|\mathbf{x}^{(s)}) \tag{11}\]

**Labeled MSVED:** When the complete form of \(\mathbf{x}^{(s)}\), \(\mathbf{y}^{(t)}\), and \(\mathbf{x}^{(t)}\) is observed in our training data, the graphical model of the labeled MSVED model is illustrated in Fig. 2 (d). We maximize the variational lower bound on the conditional log likelihood of observing \(\mathbf{x}^{(t)}\) and \(\mathbf{y}^{(t)}\) as follows:

\[\log p_{\theta}(\mathbf{x}^{(t)},\mathbf{y}^{(t)}|\mathbf{x}^{( s)})\] \[\geq\mathbb{E}_{\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x}^{( s)})}\log\frac{p_{\theta}(\mathbf{x}^{(t)},\mathbf{y}^{(t)},\mathbf{z}|\mathbf{x}^{(s)})}{q_{ \phi}(\mathbf{z}|\mathbf{x}^{(s)})}\] \[=\mathbb{E}_{\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x}^{(s)})}[ \log p_{\theta}(\mathbf{x}^{(t)}|\mathbf{y}^{(t)},\mathbf{z})+\log p_{\pi}( \mathbf{y}^{(t)})]-\] \[\text{KL}(q_{\phi}(\mathbf{z}|\mathbf{x}^{(s)})||p(\mathbf{z}))= \mathcal{L}_{l}(\mathbf{x}^{(t)},\mathbf{y}^{(t)}|\mathbf{x}^{(s)}) \tag{12}\]

## 4 Learning MSVED

Now that we have described our overall model, we discuss details of the learning process that prove 

useful to its success.

### Learning Discrete Latent Variables

One challenge in training our model is that it is not trivial to perform back-propagation through discrete random variables, and thus it is difficult to learn in the models containing discrete tags such as MSVAE or MSVED.4 To alleviate this problem, we use the recently proposed Gumbel-Softmax trick (Maddison et al., 2014; Gumbel and Lieblein, 1954) to create a differentiable estimator for categorical variables.

Footnote 4: Kingma et al. (2014) solve this problem by marginalizing over all labels, but this is infeasible in our case where we have an exponential number of label combinations.

The Gumbel-Max trick (Gumbel and Lieblein, 1954) offers a simple way to draw samples from a categorical distribution with class probabilities \(\pi_{1},\pi_{2},\cdots\) by using the argmax operation as follows: one\(\_\text{hot}(\arg\max_{i}[g_{i}+\log\pi_{i}])\), where \(g_{1},g_{2},\cdots\) are i.i.d. samples drawn from the Gumbel(0,1) distribution.5 When making inferences on the morphological labels \(y_{1},y_{2},\cdots\), the Gumbel-Max trick can be approximated by the continuous softmax function with temperature \(\tau\) to generate a sample vector \(\mathbf{\hat{y}}_{i}\) for each label \(i\):

Footnote 5: The Gumbel (0,1) distribution can be sampled by first drawing \(u\sim\) Uniform(0,1) and computing \(g=-\log(-\log(u))\).

\[\hat{y}_{ij}=\frac{\exp((\log(\pi_{ij})+g_{ij})/\tau)}{\sum_{k=1}^{N_{i}}\exp ((\log(\pi_{ik})+g_{ik})/\tau} \tag{13}\]

where \(N_{i}\) is the number of classes of label \(i\). When \(\tau\) approaches zero, the generated sample \(\mathbf{\hat{y}}_{i}\) becomes a one-hot vector. When \(\tau>0\), \(\mathbf{\hat{y}}_{i}\) is smooth w.r.t \(\pi_{i}\). In experiments, we start with a relatively large temperature and decrease it gradually.

### Learning Continuous Latent Variables

MSVED aims at generating the target sequence conditioned on the latent variable \(\mathbf{z}\) and the target labels \(\mathbf{y}^{(t)}\). This requires the encoder to generate an informative representation \(\mathbf{z}\) encoding the content of the \(\mathbf{x}^{(s)}\). However, the variational lower bound in our loss function contains the KL-divergence between the approximate posterior \(q_{\phi}(\mathbf{z}|\mathbf{x})\) and the prior \(p(\mathbf{z})\), which is relatively easy to learn compared with learning to generate output from a latent representation. We observe that with the vanilla implementation the KL cost quickly decreases to near zero, setting \(q_{\phi}(\mathbf{z}|\mathbf{x})\) equal to standard normal distribution. In this case, the RNN decoder can easily rely on the true output of last time step during training to decode the next token, which degenerates into an RNN language model. Hence, the latent variables are ignored by the decoder and cannot encode any useful information. The latent variable \(\mathbf{z}\) learns an undesirable distribution that coincides with the imposed prior distribution but has no contribution to the decoder. To force the decoder to use the latent variables, we take the following two approaches which are similar to Bowman et al. (2016).

**KL-Divergence Annealing:** We add a coefficient \(\lambda\) to the KL cost and gradually anneal it from zero to a predefined threshold \(\lambda_{\mathbf{m}}\). At the early stage of training, we set \(\lambda\) to be zero and let the model first figure out how to project the representation of the source sequence to a roughly right point in the space and then regularize it with the KL cost. Although we are not optimizing the tight variational lower bound, the model balances well between generation and regularization. This technique can also be seen in Kocisky et al. (2016); Miao and Blunsom (2016).

**Input Dropout in the Decoder:** Besides annealing the KL cost, we also randomly drop out the input token with a probability of \(\beta\) at each time step of the decoder during learning. The previous ground-truth token embedding is replaced with a zero vector when dropped. In this way, the RNN decoder could not fully rely on the ground-truth previous token, which ensures that the decoder uses information encoded in the latent variables.

## 5 Architecture for Morphological Reinflection

**Training details:** For the morphological reinflection task, our supervised training data consists of source \(\mathbf{x}^{(s)}\), target \(\mathbf{x}^{(t)}\), and target tags \(\mathbf{y}^{(t)}\). We test three variants of our model trained using different types of data and different loss functions. First, the single-directional supervised model (**SD-Sup**) is purely supervised: it only decodes the target word from the given source word with the loss function \(\mathcal{L}_{l}(\mathbf{x}^{(t)},\mathbf{y}^{(t)}|\mathbf{x}^{(s)})\) from Eq. 12. Second, the bi-directional supervised model (**BD-Sup**) is trained in both directions: decoding the target word from the source word and decoding the source word from the target word, which corresponds to the loss function \(\mathcal{L}_{l}(\mathbf{x}^{(t)},\mathbf{y}^{(t)}|\mathbf{x}^{(s)})+\mathcal{ L}_{u}(\mathbf{x}^{(s)}|\mathbf{x}^{(t)})\) using Eqs. 11-12. Finally, the semi-supervised model (**Semi-sup**) is trained to maxi 

mize the variational lower bounds and minimize the classification cross-entropy error of 10.

\[{\cal L}({\bf x^{(s)},x^{(t)},y^{(t)},x})=\alpha\cdot{\cal U}({\bf x})+{ \cal L}_{u}({\bf x^{(s)}|x^{(t)}})\] \[~{}~{}+{\cal L}_{l}({\bf x^{(t)},y^{(t)}|x^{(s)}})-{\cal D}({\bf x ^{(t)},y^{(t)}}) \tag{14}\]

The weight \(\alpha\) controls the relative weight between the loss from unlabeled data and labeled data.

We use Monte Carlo methods to estimate the expectation over the posterior distribution \(q({\bf z}|{\bf x})\) and \(q({\bf y}|{\bf x})\) inside the objective function 14. Specifically, we draw Gumbel noise and Gaussian noise one at a time to compute the latent variables \({\bf y}\) and \({\bf z}\).

The overall model architecture is shown in Fig. 3. Each character and each label is associated with a continuous vector. We employ Gated Recurrent Units (GRUs) for the encoder and decoder. Let \(\overrightarrow{h_{t}}\) and \(\overleftarrow{h_{t}}\) denote the hidden state of the forward and backward encoder RNN at time step \(t\). \({\bf u}\) is the hidden representation of \({\bf x^{(s)}}\) concatenating the last hidden state from both directions i.e. \([\overrightarrow{h_{T}};\overleftarrow{h_{T}}]\) where \(T\) is the word length. \({\bf u}\) is used as the input for the inference model on \({\bf z}\). We represent \(\mu({\bf u})\) and \(\sigma^{2}({\bf u})\) as MLPs and sample \({\bf z}\) from \({\cal N}(\mu({\bf u}),\text{diag}(\sigma^{2}({\bf u})))\), using \({\bf z}=\mu+\sigma\circ\epsilon\), where \(\epsilon\sim{\cal N}({\bf 0},{\bf I})\). Similarly, we can obtain the hidden representation of \({\bf x^{(t)}}\) and use this as input to the inference model on each label \({\bf y}_{i}^{(t)}\) which is also an MLP following a softmax layer to generate the categorical probabilities of target labels.

In decoding, we use 3 types of information in calculating the probability of the next character : (1) the current decoder state, (2) a tag context vector using attention Bahdanau et al. (2015) over the tag embeddings, and (3) the latent variable \({\bf z}\). The intuition behind this design is that we would like the model to constantly consider the lemma represented by \({\bf z}\), and also reference the tag corresponding to the current morpheme being generated at this point. We do not marginalize over the latent variable \({\bf z}\) however, instead we use the mode \(\mu\) of \({\bf z}\) as the latent representation for \({\bf z}\). We use beam search with a beam size of 8 to perform search over the character vocabulary at each decoding time step.

**Other experimental setups:** All hyperparameters are tuned on the validation set, and include the following: For KL cost annealing, \(\lambda_{m}\) is set to be 0.2 for all language settings. For character drop-out at the decoder, we empirically set \(\beta\) to be 0.4 for all languages. We set the dimension of character embeddings to be 300, tag label embeddings to be 200, RNN hidden state to be 256, and

Figure 3: Model architecture for labeled and unlabeled data. For the encoder-decoder model, only one direction from the source to target is given. The classification model is not illustrated in the diagram.

 

latent variable \({\bf z}\) to be 150. We set \(\alpha\) the weight for the unsupervised loss to be 0.8. We train the model with Adadelta Zeiler (2012) and use early-stop with a patience of 10.

## 6 Experiments

### Background: SIGMORPHON 2016

SIGMORPHON 2016 is a shared task on morphological inflection over 10 different morphologically rich languages. There are a total of three tasks, the most difficult of which is task 3, which requires the system to output the reinflection of an inflected word.6 The training data format in task 3 is in triples: (source word, target labels, target word). In the test phase, the system is asked to generate the target word given a source word and the target labels. There are a total of three tracks for each task, divided based the amount of supervised data that can be used to solve the problem, among which track 2 has the strictest limitation of only using data for the corresponding task. As this is an ideal testbed for our method, which can learn from unlabeled data, we choose track 2 and task 3 to test our our model's ability to exploit this data.

Footnote 6: Task 1 is inflection of a lemma word and task 2 is reinflection but also provides the source word labels.

As a baseline, we compare our results with the MED system Kann and Schutze (2016a) which achieved state-of-the-art results in the shared task. This system used an encoder-decoder model with attention on the concatenated source word and target labels. Its best result is obtained from an ensemble of five RNN encoder-decoders (**Ensemble**). To make a fair comparison with our models, which don't use ensembling, we also calculated single model results (**Single**).

All models are trained using the labeled training data provided for task 3. For our semi-supervised model (**Semi-sup**), we also leverage unlabeled data from the training and validation data for tasks 1 and 2 to train variational auto-encoders.

### Results and Analysis

From the results in Tab. 1, we can glean a number of observations. First, comparing the results of our full **Semi-sup** model, we can see that for all languages except Spanish, it achieves accuracies better than the single MED system, often by a large margin. Even compared to the MED ensembled model, our single-model system is quite competitive, achieving higher accuracies for Hungarian, Navajo, Maltese, and Arabic, as well as achieving average accuracies that are state-of-the-art.

Next, comparing the different varieties of our proposed models, we can see that the semi-supervised model consistently outperforms the bidirectional model for all languages. And similarly, the bidirectional model consistently outperforms the single direction model. From these results, we can conclude that the unlabeled data is beneficial to learn useful latent variables that can be used to decode the corresponding word.

Examining the linguistic characteristics of the models in which our model performs well provides even more interesting insights. Cotterell et al. cotterell2016b estimate how often the inflection process involves prefix changes, stem-internal changes or suffix changes, the results of which are shown in Tab. 2. Among the many languages, the inflection processes of Arabic, Maltese and Navajo are relatively diverse, and contain a large amount of all three forms of inflection. By examining the experimental results together with the morphological inflection process of different languages, we found that among all the languages, Navajo, Maltese and Arabic obtain the largest gains in performance compared with the ensem

Figure 4: Performance on test data w.r.t. the percentage of suffixing inflection. Points with the same x-axis value correspond to the same language results.

 

bled MED system. To demonstrate this visually, in Fig. 4, we compare the semi-supervised MSVED with the MED single model w.r.t. the percentage of suffixing inflection of each language, showing this clear trend.

This strongly demonstrates that our model is agnostic to different morphological inflection forms whereas the conventional encoder-decoder with attention on the source input tends to perform better on suffixing-oriented morphological inflection. We hypothesize that for languages that the inflection mostly comes from suffixing, transduction is relatively easy because the source and target words share the same prefix and the decoder can copy the prefix of the source word via attention. However, for languages in which different inflections of a lemma go through different morphological processes, the infected word and the target word may differ greatly and thus it is crucial to first analyze the lemma of the inflected word before generating the corresponding the reinflection form based on the target labels. This is precisely what our model does by extracting the lemma representation \({\bf z}\) learned by the variational inference model.

### Analysis on Tag Attention

To analyze how the decoder attends to the linguistic labels associated with the target word, we randomly pick two words from the Arabic and Navajo test set and plot the attention weight in Fig. 5. The Arabic word "al-'imaratiyyatu" is an adjective which means "Emirati", and its source word in the test data is ""imaratiyinyin" 7. Both of these are declensions of "imaratiyy". The source word is singular, masculine, genitive and indefinite, while the required inflection is plural, feminine, nominative and definite. We can see from the left heat map that the attention weights are turned on at several positions of the word when generating corresponding inflections. For example, "al-" in Arabic is the definite article that marks definite nouns. The same phenomenon can also be observed in the Navajo example, as well as other languages, but due to space limitation, we don't provide detailed analysis here.

Footnote 7: [https://en.wiktionary.org/wiki/%D8%A5](https://en.wiktionary.org/wiki/%D8%A5)$d9$85$b8$A7$b8$B1$D8*A7$b2$AAA0$9$8AA 



### Analyzing Effects of Size of Unlabeled Data

From Tab. 1, we can see that semi-supervised learning always performs better than supervised learning without unlabeled data. In this section, we investigate to what extent the size of unlabeled data can help with performance. We process a German corpus from a 2017 Wikipedia dump and obtain more than 100,000 German words. These words are ranked in order of occurrence frequency in Wikipedia. The data contains a certain amount of noise since we did not apply any special processing. We shuffle all unlabeled data from both the Wikipedia and the data provided in the shared task used in previous experiments, and increase the number of unlabeled words used in learning by 10,000 each time, and finally use all the unlabeled data (more than 150,000 words) to train the model. Fig. 7 shows that the performance on the test data improves as the amount of unlabeled data increases, which implies that the unsupervised learning continues to help improve the model's ability to model the latent lemma representation even as we scale to a noisy, real, and relatively large-scale dataset. Note that the growth rate of the performance grows slower as more data is added, because although the number of unlabeled data is increasing, the model has seen most word patterns in a relatively small vocabulary.

### Case Study on Reinforced Words

In Tab. 3, we examine some model outputs on the test data from the MED system and our model respectively. It can be seen that most errors of MED and our models can be ascribed to either over-copy or under-copy of characters. In particular, from the complete outputs we observe that our model tends to be more aggressive in its changes, resulting in it performing more complicated transformations, both successfully (such as Maltesee "ndammhomi" to "tindammhiex") and unsuccessfully ("tqozzx" to "quazzejtx"). In contrast, the attentional encoder-decoder model is more conservative in its changes, likely because it is less effective in learning an abstracted representation for the lemma, and instead copies characters directly from the input.

## 7 Conclusion and Future Work

In this work, we propose a multi-space variational encoder-decoder framework for labeled sequence transduction problem. The MSVED performs well in the task of morphological reinflection, outperforming the state of the art, and further improving with the addition of external unlabeled data. Future work will adapt this framework to other sequence transduction scenarios such as machine translation, dialogue generation, question answering, where continuous and discrete latent variables can be abstracted to guide sequence generation.

#### Acknowledgments

The authors thank Jiatao Gu, Xuezhe Ma, Zihang Dai and Pengcheng Yin for their helpful discussions. This work has been supported in part by an Amazon Academic Research Award.

\begin{table}
\begin{tabular}{l l r c c c} \hline \hline
**Language** & **Src Word** & **Tgt Labels** & **Gold Tgt** & **MED** & **Ours** \\ \hline \multirow{4}{*}{Turkish} & kocama & pos=N,poss=PSS1S,case=ESS,num=SG & kocamda & kocama & kocamda \\  & yaratmaman & pos=N,case=NOM,num=SG & yartma & yaratma & yaratman \\  & bitimizde & pos=N,tenese=PST,per=1,num=SG & bittik & bititydik & bittim \\ \hline \multirow{3}{*}{Maltese} & ndammholli & pos=V,polar=NEG,tenese=PST,num=SG & tindammhiex & nadmemjethix & tindammhiex \\  & tqozhieli & pos=V,polar=NEG,tenese=PST,num=SG & tqozzx & tqozzx & qazzejtx \\  &tisikkmuhomli & pos=V,polar=POS,tenese=PST,num=PL & ssiskkmulna & tissikkmulina & tissikkmulna \\ \hline \multirow{3}{*}{Finnish} & verovapaatta & pos=ADJ,case=PRT,num=PL & verovapaita & verovappaita & verovapaita \\  & tururume & pos=V,mod=POT,tenese=PRS,num=PL & turtunenme & turtunenemme & turrunenme \\ \cline{1-1}  & sukanimin & pos=N,case=PRIV,num=PL & sukuanimitt & sukunimeitta & sukunimeitta \\ \hline \hline \end{tabular}
\end{table}
Table 3: Randomly picked output examples on the test data. Within each block, the first, second and third lines are outputs that ours is correct and MED’s is wrong, ours is wrong and MED’s is correct, both are wrong respectively.

Figure 7: Performance on the German test data w.r.t. the amount of unlabeled Wikipedia data.





## References

* [Alegria and Etxeberria2016] Inaki Alegria and Izaskun Etxeberria. 2016. Ehu at the sigmorphon 2016 shared task. a simple proposal: Grapheme-to-phoneme for inflection. _In Proceedings of the 2016 Meeting of SIGMORPHON_ .
* Bahdanau et al. [2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. _The International Conference on Learning Representations_ .
* Bayer and Osendorfer [2014] Justin Bayer and Christian Osendorfer. 2014. Learning stochastic recurrent networks. _arXiv preprint arXiv:1411.7610_ .
* Bowman et al. [2016] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. _Proceedings of CoNLL_ .
* Chahuneau et al. [2013] Victor Chahuneau, Eva Schlinger, Noah A Smith, and Chris Dyer. 2013. Translating into morphologically rich languages with synthetic phrases. Association for Computational Linguistics.
* Chung et al. [2015] Junyoung Chung, Kyle Kasner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. 2015. A recurrent latent variable model for sequential data. In _Advances in neural information processing systems_. pages 2980-2988.
* Coterell et al. [2016] R. Coterell, C. Kirov, J. Sylak-Glassman, D. Yarowsky, J. Eisner, and M. Hulden. 2016. The sigmorphon 2016 shared taskmorphological reinflection. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_.
* Darwish and Oard [2007] Kareem Darwish and Douglas W Oard. 2007. Adapting morphology for arabic information retrieval. In _Arabic Computational Morphology_, Springer, pages 245-262.
* Doersch [2016] Carl Doersch. 2016. Tutorial on variational autoencoders. _arXiv preprint arXiv:1606.05908_ .
* Durrett and DeNero [2013] Greg Durrett and John DeNero. 2013. Supervised learning of complete morphological paradigms. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Atlanta, Georgia, pages 1185-1195.
* Fabius and van Amersfoort [2014] Otto Fabius and Joost R van Amersfoort. 2014. Variational recurrent auto-encoders. _arXiv preprint arXiv:1412.6581_ .
* Faruqui et al. [2016] Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. 2016. Morphological inflection generation using character sequence to sequence learning. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, San Diego, California, pages 634-643.
* Gumbel and Lieblein [1954] Emil Julius Gumbel and Julius Lieblein. 1954. Statistical theory of extreme values and some practical applications: a series of lectures. _US Government Printing Office Washington_ .
* Johnson et al. [2016] Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viegas, Martin Wattenberg, Greg Corrado, et al. 2016. Google's multilingual neural machine translation system: Enabling zero-shot translation. _arXiv preprint arXiv:1611.04558_ .
* Kann et al. [2016] Katharina Kann, Ryan Colterell, and Hinrich Schutze. 2016. Neural multi-source morphological reinflection. _arXiv preprint arXiv:1612.06027_ .
* Kann and Schutze [2016a] Katharina Kann and Hinrich Schutze. 2016a. Med: The lmu system for the sigmorphon 2016 shared task on morphological reinflection. In _In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_. Berlin, Germany.
* Kann and Schutze [2016b] Katharina Kann and Hinrich Schutze. 2016b. Single-model encoder-decoder with explicit morphological representation for reinflection. In _In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_. Berlin, Germany.
* Kikuchi et al. [2016] Yuta Kikuchi, Graham Neubig, Royhei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 1328-1338.
* Kingma et al. [2014] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In _Advances in Neural Information Processing Systems_. Montreal, Canada, pages 3581-3589.
* Kingma and Welling [2014] D.P. Kingma and M. Welling. 2014. Auto-encoding variational bayes. In _The International Conference on Learning Representations_.
* Kocisky et al. [2016] Tomas Kocisky, Gabor Melis, Edward Grefenstette, Chris Dyer, Wang Ling, Phil Blunsom, and Karl Moritz Hermann. 2016. Semantic parsing with semi-supervised sequential autoencoders. _the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ .
* Maaloe et al. [2016] Lars Maaloe, Casper Kaae Sonderby, Soren Kaae Sonderby, and Ole Winther. 2016. Auxiliary deep generative models. _Proceedings of the 33rd International Conference on Machine Learning_ .
* Maddison et al. [2014] Chris J Maddison, Daniel Tarlow, and Tom Minka. 2014. A* sampling. In _Advances in Neural Information Processing Systems_. pages 3086-3094.



Yishu Miao and Phil Blunsom. 2016. Language as a latent variable: Discrete generative models for sentence compression. _the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ .
* Nicolai et al. (2016) Garrett Nicolai, Bradley Hauer, Adam St. Arnaud, and Grzegorz Kondrak. 2016. Morphological reinflection via discriminative string transduction. _In Proceedings of the 2016 Meeting of SIGMORPHON_ .
* Ostling (2016) Robert Ostling. 2016. Morphological reinflection with convolutional neural networks. _In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ page 23.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In _Proceedings of the 2016 Conference of The North American Chapter of the Association for Computational Linguistics (NAACL)_. pages 35-40.
* new york university abu dhabi signorphon 2016 morphological reinflection shared task submission. _In Proceedings of the 2016 Meeting of SIGMORPHON_ .
* Tokuda et al. (2002) Keiichi Tokuda, Takashi Masuko, Noborn Miyazaki, and Takao Kobayashi. 2002. Multi-space probability distribution hmm. _IEICE TRANSACTIONS on Information and Systems_ 85(3):455-464.
* Toutanova et al. (2008) Kristina Toutanova, Hisami Suzuki, and Achim Ruopp. 2008. Applying morphology generation models to machine translation. In _Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics_. pages 514-522.
* Zeiler (2012) Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. _arXiv preprint arXiv:1212.5701_ .
* Zhang et al. (2016) Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016. Variational neural machine translation. _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_ .