Venue: EMNLP 2017

# Cross-lingual, Character-Level Neural Morphological Tagging

Ryan Cotterell\({}^{2}\) and Georg Heigold\({}^{3}\)

\({}^{0}\)Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA \({}^{16}\)German Research Center for Artificial Intelligence, Saarbrucken, SL, GER ryan.cotterell@jhu.edu georg.heigold@gmail.com

###### Abstract

Even for common NLP tasks, sufficient supervision is not available in many languages--morphological tagging is no exception. In the work presented here, we explore a transfer learning scheme, whereby we train character-level recurrent neural taggers to predict morphological tagings for high-resource languages and low-resource languages together. Learning joint character representations among multiple related languages successfully enables knowledge transfer from the high-resource languages to the low-resource ones, improving accuracy by up to 30%.

## 1 Introduction

State-of-the-art morphological taggers require thousands of annotated sentences to train. For the majority of the world's languages, however, sufficient large-scale annotation is not available and obtaining it would often be infeasible. Accordingly, an important road forward in low-resource NLP is the development of methods that allow for the training of high-quality tools from smaller amounts of data. In this work, we focus on transfer learning--we train a recurrent neural tagger for a low-resource language jointly with a tagger for a related high-resource language. Forcing the models to share character-level features among the languages allows large gains in accuracy when tagging the low-resource languages, while maintaining (or even improving) accuracy on the high-resource language.

Recurrent neural networks constitute the state of the art for a myriad of tasks in NLP, e.g., multilingual part-of-speech tagging Plank et al. (2016), syntactic parsing Dyer et al. (2015); Zeman et al. (2017), morphological paradigm completion Cotterell et al. (2016, 2017) and language modeling Sundermeyer et al. (2012); Melis et al. (2017); recently, such models have also improved morphological tagging Heigold et al. (2016, 2017). In addition to increased performance over classical approaches, neural networks also offer a second advantage: they admit a clean paradigm for multi-task learning. If the learned representations for all of the tasks are embedded jointly into a shared vector space, the various tasks reap benefits from each other and often performance improves for all Collobert et al. (2011b). We exploit this idea for language-to-language transfer to develop an approach for cross-lingual morphological tagging.

We experiment on 18 languages taken from four different language families. Using the Universal Dependencies treebeanks, we emulate a low-resource setting for our experiments, e.g., we attempt to train a morphological tagger for Catalan using primarily data from a related language like Spanish. Our results demonstrate the successful transfer of morphological knowledge from the high-resource languages to the low-resource languages without relying on an externally acquired bilingual lexicon or bitext. We consider both the single- and multi-source transfer case and explore how similar two languages must be in order to enable high-quality transfer of morphological taggers.1

Footnote 1: While we only experiment with languages in the same family, we show that closer languages within that family are better candidates for transfer. We remark that future work should consider the viability of more distant language pairs.

## 2 Morphological Tagging

Many languages in the world exhibit rich inflectional morphology: the form of individual words mutates to reflect the syntactic function. For example, the Spanish verb _sonar_ will appear as _sueo_ in the first person present singular, but _soiadis_ in the second person present plural, depending on the bundle of syntaco-semantic attributes associated with

the given form (in a sentential context). For concreteness, we list a more complete table of Spanish verbal inflections in Table 1. Note that some languages, e.g., Archi, Northeast Caucasian language, display a veritable cornucopia of potential forms with the size of the verbal paradigm exceeding 10,000 [23].

Standard NLP annotation, e.g., the scheme in Sylak-Glassman et al. (2015), marks forms in terms of _universal_ key-attribute pairs, e.g., the first person present singular is represented as \([{\tt pos=V},{\tt per=1},{\tt num=sg},{\tt tns=pres}]\). This bundle of key-attributes pairs is typically termed a morphological tag and we may view the goal of morphological tagging to label each word in its sentential context with the appropriate tag [69, 47]. As the part-of-speech (POS) is a component of the tag, we may view morphological tagging as a strict generalization of POS tagging, where we have significantly refined the set of available tags. All of the experiments in this paper make use of the universal morphological tag set available in the Universal Dependencies (UD) [80]. As an example, we have provided a Russian sentence with its UD tagging in Figure 1.

Transferring MorphologyThe transfer of morphology is arguably more dependent on the relatedness of the languages in question than other annotations in NLP, such as POS and named entity recognition (NER). POS lends itself nicely to a universal annotation scheme [10] and traditional NER is limited to a small number of cross-linguistically compliant categories, e.g., person and place. Even universal dependency arcs employ cross-lingual labels [80].

Morphology, on the other hand, typically requires more fine-grained annotation, e.g., grammatical case and tense. It is often the case that one language will make a semantic distinction in the form (or at all) that another does not. For example, the Hungarian noun overtly marks 17 grammatical cases and Slavic verbs typically distinguish two aspects through morphology, while English marks none of these distinctions. If the word form in the source language does not overtly mark a grammatical category in the target language, it is nighm impossible to expect a successful transfer. For this reason, much of our work focuses on the transfer of related languages--specifically exploring _how_ close two languages must be for a successful transfer. Note that the language-specific nature of morphology does not contradict the universality of the annotation; each language may mark a different subset of categories, i.e., use a different set of the universal keys and attributes, but there is a single, universal set, from which the key-attribute pairs are drawn. See Newmeyer (2007) for a linguistic treatment of cross-lingual annotation.

NotationWe will discuss morphological tagging in terms of the following notation. We will consider two (related) languages: a high-resource _source_ language \(\ell_{s}\) and a low-resource _target_ language \(\ell_{t}\). Each of these languages will have its own (potentially overlapping) set of morphological tags, denoted \(\mathcal{T}_{s}\) and \(\mathcal{T}_{t}\), respectively. We will work with the union of both sets \(\mathcal{T}=\mathcal{T}_{s}\cup\mathcal{T}_{t}\). An individual tag \(m_{i}=[k_{1}{=}v_{1},\ldots,k_{M}{=}v_{M}]\in\mathcal{T}\) is comprised of universal keys and attributes, i.e., the pairs \((k_{i},v_{i})\) are completely language-agnostic. In the case where a language does not mark a distinction, e.g., case on English nouns, the corresponding keys are excluded from the tag. Typically, \(|\mathcal{T}|\) is large (see Table 3). We denote the set of training sentences for the high-resource source language as \(\mathcal{D}_{s}\) and the set of training sentences for the low-resource target language as \(\mathcal{D}_{t}\). In the experimental section, we will also consider a multi-source setting

Figure 1: Example of a morphologically-tagged sentence in Russian using the annotation scheme provided in the UD dataset.

 

where we have multiple high-resource languages, but, for ease of explication, we stick to the single-source case in the development of the model.

## 3 Character-Level Neural Transfer

Our formulation of transfer learning builds on work in multi-task learning Caruana (1997); Collobert et al. (2011b). We treat each individual language as a task and train a joint model for all the tasks. We first discuss the current state of the art in morphological tagging: a character-level recurrent neural network. After that, we explore three augmentations to the architecture that allow for the transfer learning scenario. All of our proposals force the embedding of the characters for both the source and the target language to share the same vector space, but involve different mechanisms, by which the model may learn language-specific features.

### Character-Level Neural Networks

Character-level neural networks currently constitute the state of the art in morphological tagging Heigold et al. (2017). We draw on previous work in defining a conditional distribution over taggings \({\mathbf{t}}\) for a sentence \({\mathbf{w}}\) of length \(|{\mathbf{w}}|=N\) as

\[p_{\mathbf{\theta}}({\mathbf{t}}\mid{\mathbf{w}})=\prod_{i=1}^{N}p_{\mathbf{\theta}}(t_{i} \mid{\mathbf{w}}), \tag{1}\]

which may be seen as a \(0^{\text{th}}\) order conditional random field (CRF) Lafferty et al. (2001) with parameter vector \({\mathbf{\theta}}\).2 Importantly, this factorization of the distribution \(p_{\mathbf{\theta}}({\mathbf{t}}\mid{\mathbf{w}})\) also allows for efficient exact decoding and marginal inference in \(\mathcal{O}(|\mathcal{T}|\cdot N)\)-time, but at the cost of not admitting any explicit interactions in the output structure, i.e., between adjacent tags.3 We parameterize the distribution over tags at each time step as

Footnote 2: The parameter vector \({\mathbf{\theta}}\) is a vectorization of all the parameters discussed below.

Footnote 3: As an aside, it is quite interesting that a model with the factorization in Equation (1) outperforms the MarMoT model Müller et al. (2013), which focused on modeling higher-order interactions between the morphological tags, e.g., they employ up to a (pruned) \(3^{\text{rd}}\) order CRF. That such a model achieves state-of-the-art performance indicates, however, that richer source-side features, e.g., those extracted by our character-level neural architecture, are more important for morphological tagging than higher-order tag interactions, which come with the added unpleasantness of exponential (in the order) decoding.

\[p_{\mathbf{\theta}}(t_{i}\mid{\mathbf{w}})=\text{softmax}\left(W{\mathbf{e}}_{i}+{\mathbf{b}} \right), \tag{2}\]

where \(W\in\mathbb{R}^{|\mathcal{T}|\times n}\) is an embedding matrix, \({\mathbf{b}}\in\mathbb{R}^{|\mathcal{T}|}\) is a bias vector and positional embeddings \({\mathbf{e}}_{i}\)4 are taken from a concatenation of the output of two long short-term memory recurrent neural networks (LSTMs) Hochreiter and Schmidhuber (1997), folded forward and backward, respectively, over a sequence of input vectors. This constitutes a bidirectional LSTM Graves and Schmidhuber (2005). We define the positional embedding vector as follows

Footnote 4: Note that \(|{\mathbf{e}}_{i}|=n\); see §4.4 for the exact values used in the experimentation.

\[{\mathbf{e}}_{i}=\left[\text{LSTM}({\mathbf{v}}_{1::i});\text{LSTM}({\mathbf{v}}_{N,i+1 })\right], \tag{3}\]

where each \({\mathbf{v}}_{i}\in\mathbb{R}^{n}\) is, itself, a word embedding. Note that the function LSTM returns the _last_ final hidden state vector of the network. This architecture is the _context_ bidirectional recurrent neural network of Plank et al. (2016). Finally, we derive each word embedding vector \({\mathbf{v}}_{i}\) from a character-level bidirectional LSTM embedder. Namely, we define each word embedding as the concatenation

\[{\mathbf{v}}_{i}=\Big{[}\text{LSTM}\left(\langle c_{i_{1}},\ldots,c_{i_ {M_{i}}}\rangle\right);\\ \text{LSTM}\left(\langle c_{i_{M_{i}}},\ldots,c_{i_{1}}\rangle \right)\Big{]}\,.\]

In other words, we run a bidirectional LSTM over the character stream. This bidirectional LSTM is the _sequence_ bidirectional recurrent neural network of Plank et al. (2016). Note a concatenation of the sequence of character symbols \(\langle c_{i_{1}},\ldots,c_{i_{M_{i}}}\rangle\) results in the word string \(w_{i}\). Each of the \(M_{i}\) characters \(c_{i_{k}}\) is a member of the set \(\Sigma\). We take \(\Sigma\) to be the union of sets of characters in the languages considered.

We direct the reader to Heigold et al. (2017) for a more in-depth discussion of this and various additional architectures for the computation of \({\mathbf{v}}_{i}\); the architecture we have presented in Equation (5) is competitive with the best performing setting in Heigold et al.'s study.

### Cross-Lingual Morphological Transfer as Multi-Task Learning

Cross-lingual morphological tagging may be formulated as a multi-task learning problem. We seek to learn a set of shared character embeddings for taggers in both languages together through optimization of a joint loss function that combines the high-resource tagger and the low-resource one. The first loss function we consider is the following: 

\[{\cal L}_{multi}(\mathbf{\theta})= \sum_{(\mathbf{t},\mathbf{w})\in{\cal D}_{s}}\!\log p_{\mathbf{\theta}}(\mathbf{t} \mid\mathbf{w},\ell_{s}) \tag{5}\] \[+\sum_{(\mathbf{t},\mathbf{w})\in{\cal D}_{t}}\!\log p_{\mathbf{\theta}}(\mathbf{ t}\mid\mathbf{w},\ell_{t})\,.\]

Crucially, our cross-lingual objective forces both taggers to share part of the parameter vector \(\mathbf{\theta}\), which allows it to represent morphological regularities between the two languages in a common embedding space and, thus, enables transfer of knowledge. This is no different from monolingual multitask settings, e.g., jointly training a chunker and a tagger for the transfer of syntactic information Collobert et al. (2011b). We point out that, in contrast to our approach, almost all multi-task transfer learning, e.g., for dependency parsing Guo et al. (2016), has shared word-level embeddings rather than character-level embeddings. See SS6 for a more complete discussion.

We consider two parameterizations of this distribution \(p_{\mathbf{\theta}}(t_{i}\mid\mathbf{w},\ell)\). First, we modify the initial character-level LSTM embedding such that it also encodes the identity of the language. Second, we modify the softmax layer, creating a language-specific softmax.

Language-Universal Softmax.Our first architecture has one softmax, as in Equation (2), over all morphological tags in \({\cal T}\) (shared among all the languages). To allow the architecture to encode morphological features specific to one language, e.g., the third person present plural ending in Spanish is _-an_, but _-ao_ in Portuguese, we modify the creation of the character-level embeddings. Specifically, we augment the character alphabet \(\Sigma\) with a distinguished symbol that indicates the language: id\({}_{\ell}\). We then pre- and postpend this symbol to the character stream for every word before feeding the characters into the bidirectional LSTM Thus, we arrive at the new _language-specific_ word embeddings,

\[\mathbf{v}^{\ell}_{i}= \left[\text{LSTM}\left(\left(\texttt{id}_{\ell},c_{i_{1}}, \ldots,c_{i_{M_{l}}},\texttt{id}_{\ell}\right)\right);\right. \tag{6}\] \[\left.\hskip 85.35826pt\text{LSTM}\left(\left<\texttt{id}_{ \ell},c_{i_{M_{l}}},\ldots,c_{i_{1}},\texttt{id}_{\ell}\right>\right)\right].\]

This model creates a language-specific embedding vector \(\mathbf{v}_{i}\), but the individual embeddings for a given character are shared among the languages jointly trained on. The remainder of the architecture is held constant.

Language-Specific Softmax.Next, inspired by the architecture of Heigold et al. (2013), we consider a language-specific softmax layer, i.e., we define a new output layer for every language:

\[p_{\mathbf{\theta}}\left(t_{i}\mid\mathbf{w},\ell\right)=\text{softmax}\left(W_{\ell }\mathbf{e}_{i}+\mathbf{b}_{\ell}\right), \tag{7}\]

where \(W_{\ell}\in\mathbb{R}^{|{\cal T}|\times n}\) and \(\mathbf{b}_{\ell}\in\mathbb{R}^{|{\cal T}|}\) are now _language-specific_. In this architecture, the embeddings \(\mathbf{e}_{i}\) are the same for all languages--the model has to learn language-specific behavior exclusively through the output softmax of the tagging LSTM.

Joint Morphological Tagging and Language Identification.The third model we exhibit is a joint architecture for tagging and language identification. We consider the following loss function:

\[{\cal L}_{joint}(\mathbf{\theta})=\sum_{(\mathbf{t},\mathbf{w})\in{\cal D}_{ s}}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\! 

Just as before, we define \(p_{\mathbf{\theta}}\left(\mathbf{t}~{}|~{}\mathbf{w},\ell\right)\) above as in Equation (7) and we define

\[p_{\mathbf{\theta}}(\ell~{}|~{}\mathbf{w})=\text{softmax}\left(\mathbf{u}^{\top}\tanh(V\mathbf{ e}_{i})\right), \tag{10}\]

which is a multi-layer perceptron with a binary softmax (over the two languages) as an output layer; we have added the additional parameters \(V\in\mathbb{R}^{|T|\times n}\) and \(\mathbf{u}\in\mathbb{R}^{|\mathcal{T}|}\). In the case of multi-source transfer, this is a softmax over the set of languages.

Comparative Discussion.The first two architectures discussed in SS3.2 represent two possibilities for a multi-task objective, where we condition on the language of the sentence. The first integrates this knowledge at a lower level and the second at a higher level. The third architecture discussed in SS3.2 takes a different tack--rather than conditioning on the language, it predicts it. The joint model offers one interesting advantage over the two architectures proposed. Namely, it allows us to perform a morphological analysis on a sentence where the language is unknown. This effectively alleviates an early step in the NLP pipeline, where language id is performed and is useful in conditions where the language to be tagged may not be known _a-priori_, e.g., when tagging social media data.

While there are certainly more complex architectures one could engineer for the task, we believe we have found a relatively diverse sampling, enabling an interesting experimental comparison. Indeed, it is an important empirical question which architectures are most appropriate for transfer learning. Since transfer learning affords the opportunity to reduce the sample complexity of the "data-hungry" neural networks that currently dominate NLP research, finding a good solution for cross-lingual transfer in state-of-the-art neural models will likely be a boon for low-resource NLP in general.

## 4 Experiments

Empirically, we ask three questions of our architectures. i) How well can we transfer morphological tagging models from high-resource languages to low-resource languages in each architecture? (Does one of the three outperform the others?) ii) How many annotated data in the low-resource language do we need? iii) How closely related do the languages need to be to get good transfer?

### Experimental Languages

We experiment with the language families: Romance (Indo-European), Northern Germanic (Indo-European), Slavic (Indo-European) and Uralic. In the Romance sub-grouping of the wider Indo-European family, we experiment on Catalan (ca), French (fr), Italian (it), Portuguese (pt), Romanian (ro) and Spanish (es). In the Northern Germanic family, we experiment on Danish (da), Norwegian (no) and Swedish (sv). In the Slavic family, we experiment on Bulgarian (bg), Czech (bg), Polish (pl), Russian (ru), Slovak (sk) and Ukrainian (uk). Finally, in the Uralic family we experiment on Estonian (et), Finnish (fi) and Hungarian (hu).

### Datasets

We use the morphological tagging datasets provided by the Universal Dependencies (UD) treebanks (the concatenation of the \(4^{\text{th}}\) and \(6^{\text{th}}\) columns of the file format) Nivre et al. (2016). We list the size of the training, development and test splits of the UD treebanks we used in Table 2. Also, we list the number of unique morphological tags in each language in Table 3, which serves as an approximate measure of the morphological complexity each language exhibits. Crucially, the data are annotated in a cross-lingquistically consistent manner, such that words in the different languages that have the same synstacto-semantic function have the same bundle of tags (see SS2 for a discussion). Potentially, further gains would be possible by using a more

\begin{table}
\begin{tabular}{l c r r l l l l l} \hline  & \multicolumn{5}{c}{**Romance**} & \multicolumn{5}{c}{**Slavic**} \\ \hline lang & train & dev & test & lang & train & dev & test \\ \hline \multirow{2}{*}{(ca)} & 13123 & 1709 & 1846 & \multirow{2}{*}{(bg)} & \multirow{2}{*}{8907} & \multirow{2}{*}{1115} & \multirow{2}{*}{1116} \\  & (es) & 14187 & 1552 & 274 & & & & \\  & (ft) & 14554 & 1596 & 298 & & & & \\  & (tt) & 12837 & 489 & 489 & & & & \\  & (pr) & 8800 & 271 & 288 & & & & \\  & (ro) & 7141 & 1191 & 1191 & & & & \\  & & & & & & & & \\ \hline \end{tabular} \begin{tabular}{l c c c c c c c} \hline \multicolumn{5}{c}{**Uralic**} \\ \hline lang & train & dev & test & lang & train & dev & test \\ \hline \multirow{2}{*}{(da)} & 4868 & 322 & 322 & \multirow{2}{*}{(ec)} & \multirow{2}{*}{(e)} & \multirow{2}{*}{14510} & \multirow{2}{*}{1793} & \multirow{2}{*}{1806} \\  & (no) & 15696 & 2410 & 1939 & & & & & \\  & (su) & 4303 & 504 & 1219 & & & & & \\ \hline \end{tabular} \begin{tabular}{l c c c c c c} \hline \multicolumn{5}{c}{**H**(to)} \\ \hline \hline \multicolumn{5}{c}{**L**(to)} \\ \hline \hline \end{tabular} \begin{tabular}{l c c c c c} \hline \multicolumn{5}{c}{**Domance**} & \multicolumn{5}{c}{**Slavie**} \\ \hline lang & \(|\mathcal{T}|\) & lang & \(|\mathcal{T}|\) & lang & \(|\mathcal{T}|\) & lang & \(|\mathcal{T}|\) \\ \hline \multirow{2}{*}{(co)} & 172 & \multirow{2}{*}{(bg)} & 380 & \multirow{2}{*}{(da)} & 124 & \multirow{2}{*}{(est)} & \multirow{2}{*}{654} \\  & (es) & 232 & \multirow{2}{*}{(cs)} & 2282 & & & & \\  & (f) & 1442 & \multirow{2}{*}{(pu)} & 774 & & & & & \\  & (tu) & 179 & & & & & & \\  & (pv) & 375 & & & & & & \\  & (v) & 367 & & & & & & \\ \hline \end{tabular} 
\begin{tabular}{l c c c c c} \hline \multicolumn{5}{c}{**J**(to)} \\ \hline \hline \multicolumn{5}{c}{**L**(to)} \\ \hline \hline \end{tabular}
\end{table}
Table 3: Number of unique morphological tags for each of the experimental languages (organized by language family).

 

universal scheme, e.g., the UniMorph scheme.

### Baselines

We consider two baselines in our work. First, we consider the MarMoT tagger (Muller et al., 2013), which is currently the best performing non-neural model. The source code for MarMoT is freely available online,5 which allows us to perform fully controlled experiments with this model. Second, we consider the alignment-based projection approach of Buys and Botha (2016).6 We discuss each of the two baselines in turn.

Footnote 5: [http://cistern.cis.lmu.de/marnot/](http://cistern.cis.lmu.de/marnot/)

Footnote 6: We do not have access to the code as the model was developed in industry, so we compare to the numbers reported in the original paper, as well as additional numbers provided to us by the first author in a personal communication. The numbers will not be, strictly speaking, comparable. However, we hope they provide insight into the relative performance of the tagger.

#### 4.3.1 Higher-Order CRF Tagger

The MarMoT tagger is the leading non-neural approach to morphological tagging. This baseline is important since non-neural, feature-based approaches have been found empirically to be more efficient, in the sense that their learning curves tend to be steeper. Thus, in the low-resource setting we would be remiss to not consider a feature-based approach. Note that this is not a transfer approach, but rather only uses the low-resource data.

#### 4.3.2 Alignment-based Projection

The projection approach of Buys and Botha (2016) provides an alternative method for transfer learning. The idea is to construct pseudo-annotations for bitext given cross-lingual alignments Och and Ney (2003). Then, one trains a standard tagger using the projected annotations. The specific tagger employed is the wsabie model of Weston et al. (2011), which--like our approach-- is a \(0^{\text{th}}\)-order discriminative neural model. In contrast to ours, however, their network is shallow. We compare the two methods in more detail in SS6.

#### 4.3.3 Architecture Study

Additionally, we perform a thorough study of the neural transfer learner, considering all three architectures. A primary goal of our experiments is to determine which of our three proposed neural transfer techniques is superior. Even though our experiments focus on morphological tagging, these architectures are more general in that they may be easily applied to other tasks, e.g., parsing or machine translation. We additionally explore the viability of multi-source transfer, i.e., the case where we have multiple source languages. All of our architectures generalize to the multi-source case without any complications.

### Experimental Details

We train our models with the following conditions.

Evaluation Metrics.We evaluate using average per token accuracy, as is standard for both POS tagging and morphological tagging, and per feature \(F_{1}\) as employed in Buys and Botha (2016). The per feature \(F_{1}\) calculates a key \(F_{1}^{k}\) for each key in the target language's tags by asking if the key-attribute pair \(k_{i}{=}v_{i}\) is in the predicted tag. Then, the key-specific \(F_{1}^{k}\) values are averaged equally. Note that \(F_{1}\) is a more flexible metric as it gives partial credit for getting some of the attributes in the bundle correct, where accuracy does not.

Hyperparameters.Our networks are four layers deep (two LSTM layers for the character embedder, i.e., to compute \(\mathbf{v}_{i}\) and two LSTM layers for the tagger, i.e., to compute \(\mathbf{e}_{i}\)) and we use an embedding size of 128 for the character input vector size and hidden layers of 256 nodes in all other cases. All networks are trained with the stochastic gradient method RMSProp (Tieleman and Hinton, 2012), with a fixed initial learning rate and a learning rate decay that is adjusted for the other languages according to the amount of training data. The batch size is always 16. Furthermore, we use dropout (Srivastava et al., 2014). The dropout probability is set to 0.2. We used Torch 7 (Collobert et al., 2011) to configure the computation graphs implementing the network architectures.

## 5 Results and Discussion

We report our results in three tables. First, we report a detailed cross-lingual evaluation in Table 4. Secondly, we report a comparison against two baselines in Table 5 (accuracy) and Table 6 (\(F_{1}\)). We see two general trends of the data. First, we find that genetically closer languages yield better source languages. Second, we find that the multi-softmax architecture is the best in terms of transfer ability, as evinced by the results in Table 4. We find a wider gap between our model and the baselines under the accuracy than under \(F_{1}\). We attribute this 

to the fact that \(F_{1}\) is a softer metric in that it assigns credit to partially correct guesses.

Source Language.As discussed in SS2, the transfer of morphology is language-dependent. This intuition is borne out in the results from our study (see Table4). We see that in the closer grouping of the Western Romance languages, i.e., Catalan, French, Italian, Portuguese, and Spanish, it is easier to transfer than with Romanian, an Eastern Romance language. Within the Western grouping, we see that the close pairs, e.g., Spanish and Portuguese, are amenable to transfer. We find a similar pattern in the other language families, e.g., Russian is the best source language for Ukrainian, Czech is the best language source for Slovak and Finnish is the best source language for Estonian.

Multi-Source Transfer.In many cases, we find that multiple sources noticeably improve the results over the single-source case. For instance, when we have multiple Romance languages as a source language, we see gains of up to 2%. We also see gains in the Northern Germanic languages when using multiple source languages. From a linguistic point of view, this is logical as different source languages may be similar to the target language along different dimensions, e.g., when transferring among the Slavic languages, we note that Russian retains the complex nominal case system of Serbian, but south Slavic Bulgarian is lexically more similar.

Performance Against the Two Baselines.As shown in Table5 and Table6, our model outperforms the projection tagger of Buys and Botha (2016) even though our approach does not utilize bitext, large-scale alignment or monolingual corpora--rather, all transfer between languages happens through the forced sharing of character-level features.7 Our model, does, however, require

\begin{table}
\begin{tabular}{c c r r r r r r r r r r r r r r r r r} \hline \hline  & & & \multicolumn{8}{c}{target language} \\ \cline{3-19}  & & & \multicolumn{8}{c}{\(|\mathcal{D}_{t}|=100\)} & \multicolumn{8}{c}{\(|\mathcal{D}_{t}|=1000\)} \\ \cline{3-19}  & & & **[**]** (ca) & **[**]** (cs) & **[**]** (fr) & **[**]** (it) & **[**]** (it) & **[**]** (pt) & **[**]** (co) & **[**]** (s) & **[**]** (se) & **[**]** (f) & **[**]** (it) & **[**]** (it) & **[**]** (ro) \\ \hline \multirow{4}{*}{**Sven**} & **[**]** & **[**]** (ca) & — & 87.9\% & 84.2\% & 84.6\% & 81.1\% & 67.4\% & — & 94.1\% & **93.5\%** & 93.1\% & **89.0\%** & 89.8\% \\  & **[**]** & **[**]** (cs) & 88.9\% & — & 85.5\% & 85.6\% & 81.8\% & 69.5\% & **—** & **93.5\%** & 93.5\% & 88.9\% & 89.7\% \\  & **[**]** & **[**]** (tr) & 88.3\% & 87.0\% & — & 83.6\% & 79.5\% & **69.9\%** & 95.4\% & 93.8\% & — & 93.3\% & 88.6\% & 89.7\% \\  & **[**]** & **[**]** (it) & 88.4\% & 87.8\% & 84.2\% & — & 80.6\% & 69.1\% & 95.4\% & 94.0\% & 93.3\% & — & 88.7\% & **90.3\%** \\  & **[**]** & **[**]** (p) & 88.4\% & 88.9\% & 85.1\% & 84.7\% & — & 69.6\% & 95.3\% & **94.2\%** & **93.5\%** & 93.6\% & — & 89.8\% \\  & **[**]** & **[**]** (ro) & 87.6\% & 87.2\% & 85.0\% & 84.4\% & 79.9\% & — & 95.3\% & 93.6\% & 93.4\% & 93.2\% & 88.5\% & — \\ \hline \multirow{4}{*}{**Sven**} & multi-source & **89.8\%** & **90.9\%** & **86.6\%** & **86.6\%** & **83.4\%** & 67.5\% & 95.4\% & **94.2\%** & 93.4\% & **93.8\%** & 88.7\% & 88.9\% \\ \cline{2-19}  & & & & & & & & & & & & & & & & & & & & & \\ \hline \end{tabular}
\end{table}
Table 4: Results for transfer learning with our joint model. The tables highlight that the best source languages are often genetically and typologically closest. Also, we see that multi-source often helps, albeit more often in the \(|\mathcal{D}_{t}|=100\) case.

 

annotation of a small number of sentences in the target language for training. We note, however, that this does not necessitate a large number of human annotation hours Garrette and Baldridge (2013).

Footnote 4: [https://github.com/gurrett-babh/](https://github.com/gurrett-babh/)

Reducing Sample Complexity.Another interesting a point about our model that is best evinced in Figure 3 is the feature-based CRF approach seems to be a better choice for the low-resource setting, i.e., the neural model has greater sample complexity. However, in the multi-task scenario, we find that the neural tagger's learning curve is even steeper. In other words, if we have to train a tagger on very little data, we are better off using a neural multi-task approach than a feature-based approach; preliminary attempts to develop a multi-task version of MarMoT failed (see Figure 3).

## 6 Related Work

We divide the discussion of related work topically into three parts for ease of intellectual digestion.

### Alignment-Based Distant Supervision.

Most cross-lingual work in NLP--focusing on morphology or otherwise--has concentrated on indirect supervision, rather than transfer learning. The goal in such a regime is to provide noisy labels for training the tagger in the low-resource language through annotations projected over aligned bitext with a high-resource language. This method of projection was first introduced by Yarowsky and Ngai (2001) for the projection of POS annotation. While follow-up work Fossum and Abney (2005); Das and Petrov (2011); Tackstrom et al. (2012) has continually demonstrated the efficacy of projecting simple part-of-speech annotations, Buys and Botha (2016) were the first to show the use of bitext-based projection for the training of a _morphological_ tagger for low-resource languages.

As we also discuss the training of a morphological tagger, our work is most closely related to Buys and Botha (2016) in terms of the task itself. We contrast the approaches. The main difference lies therein, that our approach is not projection-based and, thus, does not require the construction of a bilingual lexicon for projection based on bitext.

Figure 3: Learning Curve for Spanish and Catalan comparing our monolingual model, our joint model and two MarMoT models. The first MarMoT model is identical to those trained in the rest of the paper and the second attempts a multi-task approach, which failed so no further experimentation was performed with this model.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c} \hline  & & & \multicolumn{8}{c}{Accuracy} \\  & & B\&B (2016) & & MarMoT & Ours (Mono) & Ours (Universal) & Ours (Joint) & Ours (Specific) \\ \cline{2-17}  & en (int) & best (non) & best (int) & \(100\) & \(100\) & \(100\) & \(100\) & \(100\) & \(100\) & \(100\) & \(100\) & \(100\) \\ \multirow{3}{*}{(bg)} & 36.3 & 38.2 & 50.0 & 56.5 & 78.8 & 40.2 & 66.6 & 57.8 & 80.9 & 64.5 & **81.6** & 63.5 & 80.8 \\  & cs & 24.4 & 49.3 & 53.4 & 49.2 & 69.2 & 32.1 & 66.1 & 57.4 & 77.6 & 57.9 & **74.8** & 56.1 & 74.2 \\  & da & 36.6 & 46.9 & 46.9 & 75.9 & 90.9 & 45.3 & 86.6 & 77.6 & 90.1 & 87.8 & 93.9 & 89.2 & **94.3** \\  & es & 39.9 & 75.3 & 75.5 & 85.9 & 93.1 & 64.7 & 92.5 & 85.1 & 60.9 & 90.9 & **94.2** & 90.7 & **94.2** \\  & fi & 27.4 & 51.8 & 56.0 & 50.0 & 77.5 & 28.0 & 74.2 & 48.3 & 81.2 & 55.3 & 81.8 & 55.4 & 80.7 \\  & ii & 38.1 & 75.5 & 75.9 & 81.7 & 92.3 & 67.0 & 88.9 & 84.7 & 93.1 & 86.8 & **93.8** & 86.1 & 93.3 \\  & pl & 25.3 & 47.4 & 51.3 & 51.7 & 71.1 & 32.1 & 60.9 & 47.4 & **78.4** & 57.0 & 78.1 & 56.1 & 76.4 \\  & ipt & 36.6 & 71.9 & 72.2 & 77.0 & 86.3 & 61.7 & 85.6 & 80.6 & 88.7 & 83.4 & 88.7 & 82.4 & **89.1** \\  & sw & 29.3 & 44.5 & 44.5 & 69.5 & 88.3 & 46.1 & 84.2 & 75.7 & 90.0 & 77.2 & **90.9** & 78.3 & 90.7 \\ \hline \end{tabular}
\end{table}
Table 5: Comparison of our approach to various baselines for low-resource tagging under token-level accuracy. We compare on only those languages in Buys and Botha (2016). Note that tag-level accuracy was not reported in the original B&B paper, but was acquired through personal communication with the first author. All architectures presented in this work are used in their multi-source setting. The B&B and MarMoT models are single-source.

 

Rather, our method jointly learns multiple taggers and forces them to share features--a true transfer learning scenario. In contrast to projection-based methods, our procedure always requires a minimal amount of annotated data in the low-resource target language--in practice, however, this distinction is non-critical as projection-based methods without a small mount of seed target language data perform poorly Buys and Botha (2016).

### Character-level NLP.

Our work also follows a recent trend in NLP, whereby traditional word-level neural representations are being replaced by character-level representations for a myriad tasks, e.g., POS tagging dos Santos and Zadrozny (2014), parsing Ballesteros et al. (2015), language modeling Ling et al. (2015), sentiment analysis Zhang et al. (2015) as well as the tagger of Heigold et al. (2017), whose work we build upon. Our work is also related to recent work on character-level morphological generation using neural architectures Faruqui et al. (2016); Rastogi et al. (2016).

### Neural Cross-lingual Transfer in NLP.

In terms of methodology, however, our proposal bears similarity to recent work in speech and machine translation-we discuss each in turn. In speech recognition, heigold et al. (2013) train a cross-lingual neural acoustic model on five Romance languages. The architecture bears similarity to our multi-language softmax approach. Dependency parsing benefits from cross-lingual learning in a similar fashion Guo et al. (2015, 2016).

In neural machine translation Sutskever et al. (2014); Bahdanau et al. (2015), recent work Firat et al. (2016); Zoph and Knight (2016); Johnson et al. (2016) has explored the possibility of jointly train translation models for a wide variety of languages. Our work addresses a different task, but the undergirding philosophical motivation is similar, i.e., attack low-resource NLP through multi-task transfer learning. Kann et al. (2017) offer a similar method for cross-lingual transfer in morphological inflection generation.

## 7 Conclusion

We have presented three character-level recurrent neural network architectures for multi-task cross-lingual transfer of morphological taggers. We provided an empirical evaluation of the technique on 18 languages from four different language families, showing wide-spread applicability of the method. We found that the transfer of morphological taggers is an eminnently viable endeavor among related language and, in general, the closer the languages, the easier the transfer of morphology becomes. Our technique outperforms two strong baselines proposed in previous work. Moreover, we define standard low-resource training splits in UD for future research in low-resource morphological tagging. Future work should focus on extending the neural morphological tagger to a joint lemmatizer Muller et al. (2015) and evaluate its functionality in the low-resource setting.

## Acknowledgements

RC acknowledges the support of an NDSEG fellowship. Also, we would like to thank Jan Buys and Jan Botha who help us compare to the numbers reported in their paper. We are grateful to Schloss Dagstuhl Leibniz Center for Informatics for hosting the event at which work on this paper started.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c} \hline  & & & & \multicolumn{9}{c}{\(F_{1}\)} \\  & & B\&B (2016) & & MarMot & Ours (Mono) & Ours (Universal) & Ours (Joint) & Ours (Specific) \\ \cline{2-13}  & en (int) & best (non) & best (int) & \(100\) & \(100\) & \(100\) & \(100\) & \(100\) & \(100\) & \(100\) & \(100\) & \(100\) & \(100\) \\ \multirow{4}{*}{(bg)} & 51.6 & 61.9 & 65.0 & 53.7 & 74.7 & 26.0 & 68.0 & 55.1 & 77.3 & 56.6 & 77.8 & 55.1 & **78.6** \\  & cs & 55.7 & 61.6 & 64.0 & 60.8 & 80.5 & 30.9 & 65.3 & 54.5 & 66.3 & 54.7 & 66.5 & 54.6 & **67.0** \\  & da & 65.4 & 70.7 & 73.1 & 69.7 & 92.9 & 35.3 & 90.1 & 85.9 & 93.2 & 86.9 & **93.5** & 83.2 & 93.2 \\  & es & 60.7 & 74.0 & 74.6 & 82.4 & 92.6 & 55.9 & 91.4 & 88.4 & 93.6 & 89.2 & **94.1** & 87.6 & 93.8 \\  & es & 59.1 & 57.2 & 59.1 & 44.6 & 78.3 & 17.5 & 61.7 & 48.6 & 73.6 & 49.3 & **74.4** & 46.2 & 73.9 \\  & iti & 66.1 & 74.4 & 75.3 & 78.7 & 90.0 & 56.4 & 87.0 & 83.1 & 90.5 & 83.3 & **91.9** & 82.7 & 91.7 \\  & pl & 47.3 & 56.8 & 60.4 & 57.8 & 81.8 & 31.6 & 69.7 & 61.9 & 83.9 & 62.5 & **84.7** & 62.6 & 83.2 \\  & ipt & 60.2 & 69.2 & 73.1 & 67.6 & 80.0 & 42.9 & 82.0 & 77.9 & 86.3 & 78.1 & **86.5** & 71.8 & 85.7 \\  & sv & 55.1 & 72.1 & 74.6 & 69.7 & 90.2 & 44.1 & 86.4 & 82.5 & 93.2 & 83.5 & **93.7** & 82.8 & 93.4 \\ \hline \end{tabular}
\end{table}
Table 6: Comparison of our approach to various baselines for low-resource tagging under \(F_{1}\) to allow for a more complete comparison to the model of Buys and Botha (2016). All architectures presented in this work are used in their multi-source setting. The B&B and MarMoT models are single-source. We only compare on those languages used in B&B.

 



## References

* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _International Conference on Learning Representations (ICLR)_.
* Ballesteros et al. (2015) Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 349-359, Lisbon, Portugal. Association for Computational Linguistics.
* Buys and Botha (2016) Jan Buys and Jan A. Botha. 2016. Cross-lingual morphological tagging for low-resource languages. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1954-1964, Berlin, Germany. Association for Computational Linguistics.
* Caruana (1997) Rich Caruana. 1997. Multitask learning. _Machine Learning_, 28(1):41-75.
* Collobert et al. (2011a) Ronan Collobert, Koray Kavukcuoglu, and Clement Farabet. 2011a. Torch7: A Matlab-like environment for machine learning. In _BigLearn, NIPS Workshop_.
* Collobert et al. (2011b) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011b. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* Cotterell et al. (2017) Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kubler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. The CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In _Proceedings of the CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection_, Vancouver, Canada. Association for Computational Linguistics.
* Cotterell et al. (2016) Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The SIGMORPHON 2016 shared task--morphological reinflection. In _Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_, pages 10-22, Berlin, Germany. Association for Computational Linguistics.
* Das and Petrov (2011) Dipjanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL)_, pages 600-609, Portland, Oregon, USA. Association for Computational Linguistics.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL)_, pages 334-343, Beijing, China. Association for Computational Linguistics.
* Faruqui et al. (2016) Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. 2016. Morphological inflection generation using character sequence to sequence learning. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_, pages 634-643, San Diego, California. Association for Computational Linguistics.
* Firat et al. (2016) Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_, pages 866-875, San Diego, California. Association for Computational Linguistics.
* Fossum and Abney (2005) Victoria Fossum and Steven P. Abney. 2005. Automatically inducing a part-of-speech tagger by projecting from multiple source languages across aligned corpora. In _Second International Joint Conference on Natural Language Processing (IJCNLP)_, pages 862-873.
* Garrette and Baldridge (2013) Dan Garrette and Jason Baldridge. 2013. Learning a part-of-speech tagger from two hours of annotation. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_, pages 138-147, Atlanta, Georgia. Association for Computational Linguistics.
* Graves and Schmidhuber (2005) Alex Graves and Jurgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. _Neural Networks_, 18(5-6):602-610.
* Guo et al. (2015) Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2015. Cross-lingual dependency parsing based on distributed representations. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL)_, pages 1234-1244, Beijing, China. Association for Computational Linguistics.
* Guo et al. (2016) Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2016. A representation learning framework for multi-source transfer parsing. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI)_, pages 2734-2740.
* Hajic and Hadka (1998) Jan Hajic and Barbora Hadka. 1998. Tagging ineffective languages: Prediction of morphological categories for a rich structured tagset. In _Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics

(ACL-COLING)_, pages 483-490, Montreal, Quebec, Canada. Association for Computational Linguistics.
* Heigold et al. (2017) Georg Heigold, Guenter Neumann, and Josef van Genabith. 2017. An extensive empirical evaluation of character-based morphological tagging for 14 languages. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL)_, pages 505-513, Valencia, Spain. Association for Computational Linguistics.
* Heigold et al. (2016) Georg Heigold, Gunter Neumann, and Josef van Genabith. 2016. Neural morphological tagging from characters for morphologically rich languages. _CoRR_, abs/1606.06640.
* Heigold et al. (2013) Georg Heigold, Vincent Vanhoucke, Andrew Senior, Patrick Nguyen, Marc'Aurelio Ranzato, Matthieu Devin, and Jeffrey Dean. 2013. Multilingual acoustic models using distributed deep neural networks. In _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 8619-8623. IEEE.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Johnson et al. (2016) Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viegas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's multilingual neural machine translation system: Enabling zero-shot translation. _CoRR_, abs/1611.04558.
* Kann et al. (2017) Katharina Kann, Ryan Cotterell, and Hinrich Schutze. 2017. One-shot neural cross-lingual transfer for paradigm completion. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)_, Vancouver, Canada. Association for Computational Linguistics.
* Daghestanian). In _The Handbook of Morphology_, pages 455-476. Blackwell Oxford.
* Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _Proceedings of the Eighteenth International Conference on Machine Learning (ICML)_, pages 282-289.
* Ling et al. (2015) Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1520-1530, Lisbon, Portugal. Association for Computational Linguistics.
* Melis et al. (2017) Gabor Melis, Chris Dyer, and Phil Blunsom. 2017. On the state of the art of evaluation in neural language models. _arXiv preprint arXiv:1707.05589_.
* Muller et al. (2015) Thomas Muller, Ryan Cotterell, Alexander Fraser, and Hinrich Schutze. 2015. Joint lemmatization and morphological tagging with Lemming. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2268-2274, Lisbon, Portugal. Association for Computational Linguistics.
* Muller et al. (2013) Thomas Muller, Helmut Schmid, and Hinrich Schutze. 2013. Efficient higher-order CRFs for morphological tagging. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 322-332, Seattle, Washington, USA. Association for Computational Linguistics.
* Newmeyer (2007) Frederick J. Newmeyer. 2007. Linguistic typology requires crosslinking formal categories. _Linguistic Typology_, 11(1):133-157.
* Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Samp Pypsalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In _Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC)_, pages 1659-1666.
* Och and Negy (2003) Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. _Computational Linguistics_, 29(1):19-51.
* Oflazer and Kuruoz (1994) Kemal Oflazer and Ilker Kuruoz. 1994. Tagging and morphological disambiguation of Turkish text. In _Proceedings of the Fourth Conference on Applied Natural Language Processing_, pages 144-149. Association for Computational Linguistics.
* Petrov et al. (2012) Slav Petrov, Dipanjan Das, and Ryan T. McDonald. 2012. A universal part-of-speech tagset. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC)_, pages 2089-2096.
* Plank et al. (2016) Barbara Plank, Anders Sogaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 412-418, Berlin, Germany. Association for Computational Linguistics.
* Rastogi et al. (2016) Pushpendre Rastogi, Ryan Cotterell, and Jason Eisner. 2016. Weighting finite-state transductions with neural context. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_, pages 623-633, San Diego, California. Association for Computational Linguistics 

Cleero Nogueira dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In _Proceedings of the 31th International Conference on Machine Learning (ICML)_, pages 1818-1826.
* Srivastava et al. [2014] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _Journal of Machine Learning Research_, 15(1):1929-1958.
* Sundermeyer et al. [2012] Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In _13th Annual Conference of the International Speech Communication Association (INTERSPEECH)_, pages 194-197.
* Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems (NIPS)_, pages 3104-3112.
* Sylak-Glassman et al. [2015] John Sylak-Glassman, Christo Kirov, David Yarowsky, and Roger Que. 2015. A language-independent feature schema for inflectional morphology. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL)_, pages 674-680, Beijing, China. Association for Computational Linguistics.
* Tackstrom et al. [2012] Oscar Tackstrom, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_, pages 477-487, Montreal, Canada. Association for Computational Linguistics.
* Tieleman and Hinton [2012] Tijmen Tieleman and Geoff Hinton. 2012. Lecture 6.5--RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning.
* Weston et al. [2011] Jason Weston, Samy Bengio, and Nicolas Uusnier. 2011. WSABIE: scaling up to large vocabulary image annotation. In _Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI)_, pages 2764-2770.
* Yaworsky and Ngai [2001]David Yaworsky and Grace Ngai. 2001. Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora. In _Language Technologies 2001: The Second Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
* Zeman et al. [2012] Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolabti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinkova, Jan Hajic jr., Jaroslava Hlavacova, Vaclava Kettnerova, Zdenka Uresova, Jenna Kanerva, Stina Ojala, Anna Missilia, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taiji, Nazir Habash, Herman Leung, Marie-Catherine de Marmelete, Manuela Sanquinetti, Maria Simi, Hiroshi Kanayama, Valeria dePaix, Kira Droganova, Hector Martinez Alonso, Cagr Coltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marinehecse, Georg Rehn, Tolga Kayadelen, Mohammed Attia, Ali Elkahay, Zhuoran Yu, Emily Pitler, Saran Leptradit, Michael Mandul, Jeske Kirchner, Hector Fernandez Alcalde, Jana Strnadova, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonaca, Tetiana Lando, Rittinga Nitisaroj, and Josie Li. 2017. Conll 2017 shared task: Multilingual parsing from raw text to universal dependencies. In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_, pages 1-19, Vancouver, Canada. Association for Computational Linguistics.
* Zhang et al. [2015] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _Advances in Neural Information Processing Systems (NIPS)_, pages 649-657.
* Zoph and Knight [2016] Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_, pages 30-34, San Diego, California. Association for Computational Linguistics.