Venue: NAACL 2019

# A Systematic Study of Leveraging Subword Information

for Learning Word Representations

Yi Zhu, Ivan Vulic, and Anna Korhonen

Language Technology Lab

University of Cambridge

{yz568, iv250, alk23}@cam.ac.uk

###### Abstract

The use of subword-level information (e.g., characters, character n-grams, morphemes) has become ubiquitous in modern word representation learning. Its importance is attested especially for morphologically rich languages which generate a large number of rare words. Despite a steadily increasing interest in such _subword-informed word representations_, their systematic comparative analysis across typologically diverse languages and different tasks is still missing. In this work, we deliver such a study focusing on the variation of two crucial components required for subword-level integration into word representation models: 1) _segmentation of words_ into subword units, and 2) _subword composition functions_ to obtain final word representations. We propose a general framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components, also including more advanced techniques based on position embeddings and self-attention. Using the unified framework, we run experiments over a large number of subword-informed word representation configurations (60 in total) on 3 tasks (general and rare word similarity, dependency parsing, fine-grained entity typing) for 5 languages representing 3 language types. Our main results clearly indicate that there is no "one-size-fits-all" configuration, as performance is both language- and task-dependent. We also show that configurations based on unsupervised segmentation (e.g., BPE, Morfessor) are sometimes comparable to or even outperform the ones based on supervised word segmentation.

## 1 Introduction

Word representations are central to a wide variety of NLP tasks Collobert et al. (2011); Chen and Manning (2014); Jia and Liang (2016); Ammar et al. (2016); Goldberg (2017); Peters et al. (2018); Kudo (2018), _inter alia_). Standard word representation models are based on the distributional hypothesis Harris (1954) and induce representations from large unlabeled corpora using word co-occurrence statistics Mikolov et al. (2013); Pennington et al. (2014); Levy and Goldberg (2014). However, as pointed out by recent work Bojanowski et al. (2017); Vania and Lopez (2017); Pinter et al. (2017); Chaudhary et al. (2018); Zhao et al. (2018), mapping a finite set of word types into corresponding word representations limits the capacity of these models to learn beyond distributional information, which leads to several fundamental limitations.

The standard approaches ignore the internal structure of words, that is, the syntactic or semantic composition from subwords or morphemes to words, and are incapable of parameter sharing at the level of subword units. Assigning only a single vector to each word causes the data sparsity problem, especially in resource-poor settings where huge amounts of training data cannot be guaranteed. The issue is also prominent for morphologically rich languages (e.g., Finnish) with productive morphological systems that generate a large number of infrequent/rare words Gerz et al. (2018). Although potentially useful information on word relationships is hidden in their internal _subword-level structure_,1 subword-agnostic word representation models do not take these structure features into account and are effectively unable to represent rare words accurately, or unseen words at all.

Footnote 1: For example, nouns in Finnish have 15 cases and 3 plural forms; Spanish verbs may contain over 40 inflected forms, sharing the lemma and taking up standard suffixes.

Therefore, there has been a surge of interest in _subword-informed_ word representation architectures aiming to address these gaps. A large number of architectures has been proposed in related research, and they can be clustered over the two main axes Lazaridou et al. (2013); Luong et al. (2013);

Qiu et al., 2014; Cotterell and Schutze, 2015; Wieting et al., 2016; Avraham and Goldberg, 2017; Vania and Lopez, 2017; Pinter et al., 2017; Cotterell and Schutze, 2018). First, the models differ in the chosen method for _segmenting words into subwords_. The methods range from fully supervised approaches Cotterell and Schutze, 2015 to e.g. unsupervised approaches based on BPE Heinzerling and Strube, 2018). Second, another crucial aspect is the _subword composition function_ used to obtain word embeddings from the embeddings of each word's constituent subword units. Despite a steadily increasing interest in such subword-informed word representations, their systematic comparative analysis across the two main axes, as well as across typologically diverse languages and different tasks is still missing.2

Footnote 2: A preliminary study of Vanía and Lopez (2017) limits its focus on the use of subwords in the language modeling task.

In this work, we conduct a systematic study of a variety of subword-informed word representation architectures that all can be described by a general framework illustrated by Figure 1. The framework enables straightforward experimentation with prominent word segmentation methods (e.g., BPE, Morfessor, supervised segmentation systems) as well as subword composition functions (e.g., addition, self-attention), resulting in a large number of different _subword-informed configurations_.3

Footnote 3: Following a similar work on subword-agnostic word embedding learning Levy et al., 2015), our system design choices resulting in different configurations can be seen as a set of hyper-parameters that also have to be carefully tuned for each language and each application task.

Our study aims at providing answers to the following crucial questions: **Q1)** How generalizable are subword-informed models across typologically diverse languages and across different downstream tasks? Do different languages and tasks require different configurations to reach peak performances or is there a single best-performing configuration? **Q2)** How important is it to choose an appropriate segmentation and composition method? How effective are more generally applicable unsupervised segmentation methods? Is it always better to resort to a supervised method, if available? **Q3)** Is there a difference in performance with and without the full word representation? Can more advanced techniques based on position embeddings and self-attention yield better task performance?

We evaluate subword-informed word representation configurations originating from the general framework in three different tasks using standard benchmarks and evaluation protocols: 1) general and rare word similarity and relatedness, 2) dependency parsing, and 3) fine-grained entity typing for 5 languages representing 3 language families (fusional, introflexive, agglutinative). We show that different tasks and languages indeed require diverse subword-informed configurations to reach peak performance: this calls for a more careful language- and task-dependent tuning of configuration components. We also show that more sophisticated configurations are particularly useful for representing rare words, and that unsupervised segmentation methods can be competitive to supervised segmentation in tasks such as parsing or fine-grained entity typing. We hope that this paper will provide useful points of comparison and comprehensive guidance for developing next-generation subword-informed word representation models for typologically diverse languages.

## 2 Methodology

The general framework for learning subword-informed word representations, illustrated by Figure 1, is introduced in SS2.1. We then describe its main components: segmentation of words into subword units (SS2.2), subword and position embeddings (SS2.3), and subword embedding composition

Figure 1: Illustration of the general framework for learning subword-informed word representations, with the focus on two crucial components: 1) segmentation of words and 2) subword embedding composition. By varying the two components, and optionally including or excluding position embeddings from the computations, we obtain a wide spectrum of different subword-informed configurations used in the study (see §2). Our word-level representation model in this work is skipgram (on the top layer of the figure), but it can be replaced by any other distributional word-level model.

 

functions (SS2.4), along with all the configurations for these components used in our evaluation.

### General Framework

Formally, given a word \(w\), its word embedding \({\bf w}\) can be computed by the composition of its subword embeddings as follows:

\[{\bf w}=f_{\mathbf{\Theta}}(\delta(w),{\bf W}_{s},{\bf W}_{p}), \tag{1}\]

where \(\delta(w)\) is a deterministic function that segments \(w\) into an ordered sequence of its constituent subword units \(S_{w}=(s_{w_{i}})^{n}_{1}\), with \(s_{w_{i}}\in\mathcal{S}\) being a subword type from the subword vocabulary \(\mathcal{S}\) of size \(|\mathcal{S}|\). Optionally, some segmentation methods can also generate a sequence of the corresponding morphotactic tags \(T_{w}=(t_{w_{i}})^{n}_{1}\). Alone or together with \(T_{w}\), \(S_{w}\) is embedded into a sequence of subword representations \({\bf S}_{w}=({\bf s}_{w_{i}})^{n}_{1}\) from the subword embedding matrix \({\bf W}_{s}\in\mathbb{R}^{|\mathcal{S}|\times d}\), where \(d\) is the dimensionality of subword embeddings. Another optional step is to obtain a sequence of position embeddings \({\bf P}_{w}=({\bf p}_{w_{i}})^{n}_{1}\): they are taken from the position embedding matrix \({\bf W}_{p}\in\mathbb{R}^{p\times d}\), where \(p\) is the maximum number of the unique positions. \({\bf P}_{w}\) can interact with \({\bf S}_{w}\) to compute the final representations for subwords \({\bf R}_{w}=({\bf r}_{w_{i}})^{n}_{1}\)(Vaswani et al., 2017). \(f_{\mathbf{\Theta}}\) is a composition function taking \({\bf R}_{w}\) as input and outputting a single vector \({\bf w}\) as the word embedding of \(w\).

For the distributional "word-level" training, similar to prior work Bojanowski et al. (2017), we adopt the standard skip-gram with negative sampling (sgns)Mikolov et al. (2013) with bag-of-words contexts. However, we note that other distributional models can also be used under the same framework. Again, following Bojanowski et al. (2017), we calculate the word embedding \({\bf w}_{t}\in\mathbb{R}^{d}\) for each target word \(w_{t}\) using the formulation from Eq. (1), and parametrize context words with another word embedding matrix \({\bf W}_{c}\in\mathbb{R}^{|\mathcal{V}|\times d}\), where \(|\mathcal{V}|\) is the size of word vocabulary \(\mathcal{V}\).

### Segmentation of Words into Subwords

We consider three well-known segmentation methods for the function \(\delta\), briefly outlined here.

Supervised Morphological SegmentationWe use CHIPMUNK Cotterell et al. (2015) as a representative supervised segmentation system, proven to provide a good trade-off between accuracy and speed.4 It is based on semi-Markov conditional random fields Sarawagi and Cohen (2005). For each word, apart from generating \(S_{w}\), it also outputs the corresponding morphotactic tags \(T_{w}\).5 In SS2.3 we discuss how to incorporate information from \(T_{w}\) into subword representations.

Footnote 5: In our experiments, we use only basic information on affixes such as prefixes and suffixes, and leave the integration of fine-grained information such as inflectional and derivational affixes as future work.

Morfessor Smit et al. (2014) denotes a family of generative probabilistic models for unsupervised morphological segmentation used, among other applications, to learn morphologically-aware word embeddings Luong et al. (2013).

BpeByte Pair Encoding BPE Gage (1994) is a simple data compression algorithm. It has become a de facto standard for providing subword information in neural machine translation Sennrich et al. (2016). The input word is initially split into a sequence of characters, with each unique character denoted as a byte. BPE then iteratively replaces the most common pair of consecutive bytes with a new byte that does not occur within the data, and the number of iterations can be set in advance to control the granularity of the byte combinations.

An example output for all three methods is shown in Table 1. Note that a standard practice in subword-informed models is to also insert the entire word token into \(S_{w}\)Bojanowski et al. (2017).6 This is, however, again an optional step and we evaluate configurations with and without the inclusion of the word token in \(S_{w}\).

Footnote 6: We only do the insertion if \(|S_{w}|>1\). For CHIPMUNK, a generic tag word is added to the sequence \(T_{w}\).

### Subword and Position Embeddings

The next step is to encode \(S_{w}\) (or the tuple (\(S_{w}\), \(T_{w}\)) for CHIPMUNK) to construct a sequence of subword representations \({\bf S}_{w}\). Each row of the subword embedding matrix \({\bf W}_{s}\) is simply defined as the embedding of a unique subword. For CHIPMUNK, we define each row in \({\bf W}_{s}\) as the concatenation of the subword \(s\) and its predicted tag \(t\). We also test

\begin{table}
\begin{tabular}{l l} \hline \(\delta\) & \(\delta\)(_dishonestly_) \\ \hline CHIPMUNK & (_dis_, _honest_, _ly_) \\  & (_prefix_, _root_, _suffix_) \\ \hline Morfessor & (_dishonest_, _ly_) \\ \hline BPE & (_dish_, _on_, _est_, _ly_) \\ \hline \end{tabular}
\end{table}
Table 1: Segmentations of the word _dishonestly_.

 

CHIPMUNK configurations without the use of \(T_{w}\) to analyze its contribution.7

Footnote 6: The extra information on tags should lead to a more expressive model resolving subword ambiguities. For instance, the subword _post_ in _postwar_ and noun _post_ are intrinsically different: the former is the prefix and the later is the root.

After generating \(\mathbf{S}_{w}\), an optional step is to have a _learnable_ position embedding sequence \(\mathbf{P}_{w}\) further operate on \(\mathbf{S}_{w}\) to encode the order information. Similar to \(\mathbf{W}_{s}\), the definition of the position embedding matrix \(\mathbf{W}_{p}\) also varies: for Morfessor and BPE, we use the absolute positions of subwords in the sequence \(S_{w}\), whereas for CHIPMUNK morphotactic tags are encoded directly as positions.

Finally, following prior work Gehring et al. (2017); Mikolov et al. (2018), we use addition and element-wise multiplication between each subword vector \(\mathbf{s}\) from \(\mathbf{S}_{w}\) and the corresponding position vector \(\mathbf{p}\) from \(\mathbf{P}_{w}\) to compute each entry \(\mathbf{r}\) for the final sequence of subword vectors \(\mathbf{R}_{w}\):

\[\mathbf{r}=\mathbf{s}+\mathbf{p}\quad\text{or}\quad\mathbf{r}=\mathbf{s} \odot\mathbf{p}. \tag{2}\]

### Composition Functions

A composition function \(f_{\Theta}\) is then applied to the sequence of subword embeddings \(\mathbf{R}_{w}\) to compute the final word embedding w. We investigate three composition functions: 1) addition, 2) single-head and 3) multi-head self-attention Vaswani et al. (2017); Lin et al. (2017).8 Addition is used in the original fastText model of Bojanowski et al. (2017), and remains a strong baseline for many tasks. However, addition treats each subword with the same importance, ignoring semantic composition and interactions among the word's constituent subwords. Therefore, we propose to use a self-attention mechanism, that is, a learnable weighted addition as the composition function on subword sequences. To the best of our knowledge, we are the first to apply a self-attention mechanism to the problem of subword composition.

Footnote 8: Using more complex compositions based on CNNs and RNNs is also possible, but we have not observed improvement in our evaluation tasks with such compositions, which is also in line with findings from recent work Li et al. (2018).

Composition Based on Self-AttentionOur self-attention mechanism is inspired by Lin et al. (2017). It is essentially a multilayer feed-forward neural network without bias term, which generates a weight matrix for the variable length input \(\mathbf{R}_{w}\):

\[\mathbf{H}_{w} =tanh(\mathbf{W}_{h_{1}}\mathbf{R}^{T}_{w}) \tag{3}\] \[\mathbf{A}_{w} =softmax(\mathbf{W}_{h_{2}}\mathbf{H}_{w}) \tag{4}\]

Each row of \(\mathbf{A}_{w}\) is a weight vector for rows of \(\mathbf{R}_{w}\), which models different aspects of semantic compositions and interactions. For the single-head self-attention, \(\mathbf{A}_{w}\) degenerates to a row vector as the final attention vector \(\mathbf{a}_{w}\). For the multi-head self-attention, we average the rows of \(\mathbf{A}_{w}\) to generate \(\mathbf{a}_{w}\).9 Finally, \(\mathbf{w}\) is computed as the weighted addition of subword embeddings from \(\mathbf{R}_{w}\): \(\mathbf{w}=\sum_{w_{1}}^{w_{n}}a_{w_{1}}\mathbf{r}_{w_{i}}\).

Footnote 9: We have also experimented with adding an extra transformation layer over attention matrix to generate the attention vector, but without any performance gains.

## 3 Experimental Setup

We train different subword-informed model configurations on 5 languages representing 3 morphological language types: English (en), German (de), Finnish (fi), Turkish (tr) and Hebrew (he), see Table 3. We then evaluate the resulting subword-informed word embeddings in three distinct tasks: 1) general and rare word similarity and relatedness, 2) syntactic parsing, and 3) fine-grained entity typing. The three tasks have been selected in particular as they require different degrees of syntactic and semantic information to be stored in the input word embeddings, ranging from a purely semantic task (word similarity) over a hybrid syntactic-semantic task of entity typing to syntactic parsing.

Subword-Informed ConfigurationsWe train a large number of subword-informed configurations by varying the segmentation method \(\delta\) (SS2.2), subword embeddings \(\mathbf{W}_{s}\), the inclusion of position embeddings \(\mathbf{W}_{p}\) and the operations on \(\mathbf{W}_{s}\) (SS2.3), and the composition functions \(f_{\Theta}\) (SS2.4). The configurations are based on the following variations of

\begin{table}
\begin{tabular}{c l c} Component & Option & Label \\ \hline Segmentation & CHIPMUNK & _sms_ \\  & Morfesser & _mort_ \\  & BPE & _bpe_ \\ Morphotactic tag & concatenated with & _st_ \\  & subword (only _sss_) & \\ Word token & exclusion & _w-_ \\ Position embedding & exclusion & _p-_ \\  & additive & _pp_ \\  & multiplicative & _mp_ \\ Composition function & addition & _add_ \\  & single head attention & _att_ \\  & multi-head attention & _mtx_ \\ \hline \end{tabular}
\end{table}
Table 2: Different components used to construct subword-informed configurations, and their labels.

 

constituent components: (1) For the segmentation \(\delta\), we test a supervised morphological system CHIP-MUNK (_sms_), Morfessor (_morf_) and BPE (_bpe_). A word token can be optionally inserted into the subword sequence \(S_{w}\) for all three segmentation methods (_ww_) or left out (_w_-); (2) We can only embed the subword \(s\) for _morf_ and _bpe_, while with _sms_ we can optionally embed the concatenation of the subword and its morphotactic tag \(s:t\) (_st_);10 (3) We test subword embedding learning without position embeddings (_p_-), or we integrate them using addition (_pp_) or element-wise multiplication (_mp_); (4) For the composition function \(f_{\Theta}\), we experiment with addition (_add_), single head self-attention (_att_), and multi-head self-attention (_mtx_). Table 2 provides an overview of all components used to construct a variety of subword-informed configurations used in our evaluation.

Footnote 10: Once _st_ is applied, we do not use position embeddings anymore, because the morphotactic tags are already encoded in subword embeddings, i.e., _st_ and _pp_ are mutually exclusive.

The variations of components from Table 2 yield 24 different configurations in total for _sms_, and 18 for _mof_ and _bpe_. We use pretrained CHIPMUNK models for all test languages except for Hebrew, as Hebrew lacks gold segmentation data. Following Vania Lopez (2017), we use the default parameters for Morfessor, and 10k merge operations for BPE across languages. We use available BPE models pre-trained on Wikipedia by Heinzerling and Strube (2018).11

Footnote 11: [https://github.com/heinzering/bremb](https://github.com/heinzering/bremb)

Two well-known word representation models, which can also be described by the general framework from Figure 1, are used as insightful baselines: the subword-agnostic Sgns model Mikolov et al. (2013) and fastText (ft)12Bojanowski et al. (2017). ft computes the target word embedding using addition as the composition function, while the segmentation is straightforward: the model simply generates all character n-grams of length 3 to 6 and adds them to \(S_{w}\) along with the full word.

Footnote 12: [https://github.com/facebookresearch/fastText](https://github.com/facebookresearch/fastText)

Training SetupOur training data for all languages is Wikipedia. We lowercase all text and replace all digits with a generic tag #. The statistics of the training corpora are provided in Table 3.

All subword-informed variants are trained on the same data and share the same parameters for the sgns model.13 Further, we use adagradDuchi et al. (2011) with a linearly decaying learning rate, and do a grid search of learning rate and batch size for each \(\delta\) on the German14WordSim-353 data set (ws; Leviant and Reichart (2015)). The hyperparameters are then fixed for all other languages and evaluation runs. Finally, we set the learning rate to 0.05 for _sms_ and _bpe_, and 0.075 for _mof_, and the batch size to 1024 for all the settings.

Footnote 13: We rely on the standard choices: \(300\)-dimensional subword and word embeddings, 5 training epochs, the context window size is 5, 5 negative samples, the subsampling rate of \(10^{-5}\), and the minimum word frequency is 5.

### Evaluation Tasks

Word Similarity and RelatednessThese standard intrinsic evaluation tasks test the semantics of word representations Pennington et al. (2014); Bojanowski et al. (2017). The evaluations are performed using the Spearman's rank correlation score between the average of human judgement similarity scores for word pairs and the cosine similarity between two word embeddings constituting each word pair. We use Multilingual SimLex-999 (simlex; Hill et al. (2015); Leviant and Reichart (2015); Mrksic et al. (2017)) for English, German and Hebrew, each containing 999 word pairs annotated for true semantic similarity. We further evaluate embeddings on FinnSim-300 (rs300) produced by Venekoski and Vankka (2017) for Finnish and AnlamVeran; Ercan and Yildiz (2018)) for Turkish. We also run experiments on the WordSim-353 test set (ws; Finkelstein et al. (2002)), and its portions oriented towards true similarity (ws-sim) and broader relatedness (ws-rel) portion for English and German.

Finally, to analyze the importance of subword information for learning embeddings of rare words, we evaluate on the recently released CARD-660 dataset cardPilehvar et al. (2018) for English, annotated for true semantic similarity.

\begin{table}
\begin{tabular}{c c c c} \hline \hline Topology & Language & \#tokens & \#types \\ \hline \multirow{2}{*}{Fusional} & English (en) & 600M & 900K \\  & German (de) & 200M & 940K \\ \cline{2-4}  & Finnish (fi) & 66M & 600K \\ \cline{2-4}  & Turkish (tr) & 52M & 300K \\ \cline{2-4}  & Hebrew (he) & 90M & 410K \\ \hline \hline \end{tabular}
\end{table}
Table 3: Statistics of our Wikipedia training corpora. For faster training, we use one third of the entire Wikipedia corpus for en and de.

 

Dependency ParsingNext, we use the syntactic dependency parsing task to analyze the importance of subword information for syntactically-driven downstream applications. For all test languages, we rely on the standard Universal Dependencies treebanks (UD v2.2; Nivre et al. (2016)). We use subword-informed word embeddings from different configurations to initialize the deep biaffine parser of Dozat and Manning (2017) which has shown competitive performance in shared tasks Dozat et al. (2017) and among other parsing models Ma and Hovy (2017); Shi et al. (2017); Ma et al. (2018).15 We use default settings for the biaffine parser for all experimental runs

Footnote 15: [https://github.com/tdozat/Parser-v2](https://github.com/tdozat/Parser-v2)

Fine-Grained Entity TypingThe task is to map entities, which could comprise more than one entity token, to predefined entity types Yaghoobzadeh and Schutze (2015). It is a suitable semi-semantic task to test our subword models, as the subwords of entities usually carry some semantic information from which the entity types can be inferred. For example, _Lincolnshire_ will belong to /location/county as _-shire_ is a suffix that strongly indicates a location. We rely on an entity typing dataset of Heinzerling and Strube (2018) built for over 250 languages by obtaining entity mentions from Wikidata Vrandecic and Krotzsch (2014) and their associated FIGER-based entity types Ling and Weld (2012): there only exists a one-to-one mapping between the entity and one of the 112 FIGER types.

We randomly sample the data to obtain a train/dev/test split with the size of 60k/20k/20k for all languages. For evaluation we extend the RNN-based model of Heinzerling and Strube (2018), where they stacked all the subwords of entity tokens into a flattened sequence: we use the hierarchical embedding composition instead. For each entity token, we first compute its word embeddings with our subword configurations,16 then feed the word embeddings of entity tokens to a bidirectional LSTM with 2 hidden layers of size 512, followed by a projection layer which predicts the entity type.

\begin{table}
\begin{tabular}{l l c r r}  & & Best & 2nd Best & Worst \\ \cline{3-5}  & ws & **.656** (_sms.w.-pt.att_) & .655 (_sms.w.-pp.att_) & .440 (_type.w.-mp.ntx_) \\ sw-sim & w-ssim & **.708** (_sns.w.t.mtx_) & .707 (_sw.w.-st.att_) & .475 (_hw.-np.ntx_) \\ en & ws-rel & **.625** (_sms.w.-p.-odd_) & .620 (_sms.w.-attn.dt_) & .438 (_type.w.-mp.ntx_) \\ simlex & simlex & 283 (_sms.ws.wp.-add_) & .282 (_morf.w.-p.add_) & .182 (_type.w.-mp.ntx_) \\  & ws & **.633** (_sms.ww.pp.add_) & **.633** (_sms.ww.p.-add_) & .328 (_type.w.-mp.ntx_) \\  & ws-sim & .673 (_sms.ww.pp.add_) & .668 (_sms.we.p.-add_) & .363 (_type.w.-mp.ndx_) \\ de & ws-rel & **.616** (_sms.ww.p.-add_) & .610 (_sms.ww.pp.add_) & .332 (_type.w.-mp.ntx_) \\  & simllex & **.401** (_sms.ww.p.-add_) & .398 (_sms.ww.pp.add_) & .189 (_type.w.-mp.ntx_) \\ fi & fs300 & .259 (_sms.w.-p.add_) & .258 (_sms.w.-pp.add_) & .123 (_morf.ww.mp.mtx_) \\  & an-sim & **.355** (_type.ww.mp.add_) & .325 (_type.ww.mp.att_) & .112 (_type.ww.pp.ntrx_) \\ tr & an-nrl & **.459** (_type.ww.mp.add_) & .444 (_sms.w.-pp.add_) & .273 (_morf.ww.mp.att_) \\ he & simlex & .338 (_type.ww.pp.add_) & .338 (_type.wp.mpnx_) & .128 (_type.w.-mp.ntx_) \\ en & card & **.370** (_sms.ww.pp.add_) & .328 (_sms.w.-pp.mtx_) & .009 (_type.w.-pp.add_) \\ \hline \end{tabular}
\end{table}
Table 4: Results on word similarity and relatedness across languages. The highest score for each row is in bold, and we choose randomly in case of a tie. All scores are obtained after computing the embeddings of OOV words.

\begin{table}
\begin{tabular}{c l c c c c}  & & Dev set & \multicolumn{2}{c}{Test set} \\ \cline{3-5}  & & UAS & LAS & UAS & LAS \\ \cline{3-5}  & _sms.w.-mp.ntx_ & **92.3** & 90.3 & 92.0 & 90.1 \\  & _bpe.ww.-pmx_ & **92.3** & **90.4** & **92.1** & 90.0 \\ EN & sgns & **92.3** & **90.4** & 91.9 & 89.8 \\  & ft & **92.3** & 90.3 & **92.1** & **90.2** \\  & _bpe.ww.pp.add_ & 91.2 & 87.7 & **89.6** & **84.7** \\  & _bpe.ww.mp.mtx_ & 91.2 & 87.7 & 89.4 & **84.7** \\ DE & sgns & 91.4 & **87.9** & 89.3 & 84.4 \\  & ft & **91.6** & **87.9** & 89.1 & 84.4 \\  & _bpe.ww.mp.add_ & **89.9** & **86.9** & **90.7** & **87.4** \\  & _sms.w.-pp.add_ & 89.3 & 86.1 & 90.5 & 87.1 \\ FI & signs & **88.9** & 85.6 & 89.5 & 86.2 \\  & ft & 89.7 & **86.9** & 90.4 & 87.1 \\  & _sms.ww.mp.add_ & 71.1 & **63.5** & 72.8 & 64.7 \\  & _sms.w.-np.att_ & 70.5 & 62.5 & 72.7 & 64.5 \\ TR & sgns & 70.5 & 62.5 & 72.2 & 63.5 \\ fit & **71.2** & 63.3 & **73.1** & **65.1** \\  & _morf.w.-pp.mtx_ & 92.3 & 89.5 & 91.3 & 88.5 \\  & _morf.w.-pp.-add_ & 92.3 & 89.5 & 91.2 & 88.5 \\ HE & sgns & 92.4 & **89.8** & **91.5** & **88.7** \\ ft & **92.6** & 89.7 & 91.2 & 88.3 \\ \hline \end{tabular}
\end{table}
Table 5: Results on the dependency parsing task. The two best configurations are selected according to LAS.

 

Results and Analysis

To get a better grasp of the overall performance without overloading the tables, we focus on reporting two best configurations and the worst configuration for each task and language from the total of 60 configurations, except for Hebrew with 36 configurations, where there is no gold segmentation data for training _sms_ model. We also analyze the effects of different configurations on different tasks based on language typology. The entire analysis revolves around the key questions Q1-Q3 posed in the introduction. The reader is encouraged to refer to the supplementary material for the complete results.

Tables 4, 5, and 6 summarize the main results on word similarity and relatedness, dependency parsing and entity typing, respectively. In addition, the comparisons of different configurations across tasks and language types are shown in Figure 2 (as well as Figure 3 to 7 in the supplementary material). There, we center the comparison around two crucial components: segmentation and composition. The value in each pixel block is the percentage rank of the row configuration minus that of column configuration. We compute such percentage ranks by performing three levels of averaging over: 1) all related datasets for the same task; 2) all sub-configurations that entail the configuration in question; 3) all languages from the same language types.

#### 4.2.3 Q1. Tasks and Languages

Regarding the absolute performance of our subword-informed configurations, we notice that they outperform sgns and ft in 3/5 languages on average, and for 8/13 datasets on word similarity and relatedness. The gains are more prominent over the subword-agnostic sgns model and for morphologically richer languages such as Finnish and Turkish. The results on the two other tasks are also very competitive, with strong performance reported especially on the entity typing task. This clearly indicates the importance of integrating subword-level information into word vectors. A finer-grained comparative analysis shows that best-performing configurations vary greatly across different languages.

The comparative analysis across tasks also suggests that there is no single configuration that outperforms the others in all three tasks, although certain patterns in the results emerge. For instance, the supervised segmentation (_sms_) is very useful for word similarity and relatedness (seen in Figure 4). This result is quite intuitive: _sms_ is trained according to the readily available gold standard morphological segmentations. However, _sms_ is less useful for entity typing, where almost all best-performing configurations are based on _morf_ (see also Figure 4). This result is also interpretable: _morf_ is a conservative segmenter that captures longer subwords, and is not distracted by short and nonsensical subwords (like _bpe_) that have no contribution to the prediction.17 The worst configurations on word similarity are based on _bpe_: its aggressive segmentation often results in non-interpretable or nonsensical subwords which are unable to recover useful semantic information. Due to the same reason, the results in all three tasks indicate that the best configurations with _bpe_ are always coupled with _ww_, and the worst are obtained with \(w\)- (i.e., without the inclusion of the full word).

Footnote 17: For example, _sms_ and _bpe_ both split “Valberg” (/location/city) and “Robert Valberg” (/person/actor) with a suffix “berg”. Since “breg” could represent both a place or a person, it is not useful alone as a suffix to predict the correct entity type, whereas _morf_ does not split the word and makes the prediction solely based on surrounding entity tokens.

The results on parsing reveal similar performances for a spectrum of heterogeneous configurations. In other words, while the chosen configuration is still important, its impact on performance of state-of-the-art dependency parsers Kiperwasser and Goldberg (2016); Dozat and Manning (2017) is decreased, as such parsers are heavily parametrized multi-component methods (e.g., besides word embeddings they rely on biLSTMs, intra-sentence attention, character representations).18 Therefore, a larger space of subword-informed configurations for word representations leads to optimal or near-optimal results. However, _sms_ seems to yield highest scores on average in agglutinative languages (see also Figure 2).

Footnote 18: We also experimented with removing token features such as POS tags and character embeddings in some settings, but we observed similar trends in the final results.

Figure 2 clearly demonstrates that, apart from entity typing heatmaps (the third row) which show very similar trends over different language types, the patterns for different tasks and language types tend to vary in general. Similarly, other figures in the supplementary material also show diverging trends across languages representing different lan



Q2 and Q3. ConfigurationsAs mentioned, different tasks reach high scores with different segmentation and composition components. The crucial components for word similarity are _sms_ and

\begin{table}
\begin{tabular}{l l l l l}  & Best & 2nd Best & Worst & sgns & ft \\ en & **55.70** (_bpe.ww.mp.add_) & 55.68 (_morf w-pp.att_) & 51.15 (_bpe.w.-p.-att_) & 51.00 & 55.15 \\ de & 54.06 (_morf w-pp.add_) & 54.01 (_morf w-pp.att_) & 50.21 (_bpe.w.-p.-att_) & 50.14 & **54.55** \\ fi & **57.41** (_morf w-pp.add_) & 57.38 (_morf w-pp.mtx_) & 52.18 (_bpe.w.-p.-att_) & 49.87 & 57.18 \\ tr & **56.31** (_morf w-pp.add_) & 56.14 (_morf w-pp.mtx_) & 46.97 (_bpe.w.-pp.mx_) & 54.35 & 54.62 \\ he & **60.34** (_morf w-pp.att_) & 60.09 (_morf w.wp.add_) & 51.31 (_bpe.w.-p.-att_) & 54.55 & 59.09 \\ \hline \end{tabular}
\end{table}
Table 6: Test accuracy, the evaluation metric used by Heinzerling and Strube (2018), on the fine-grained entity typing task. The results are averaged over 5 runs with random seeds.

Figure 2: Comparisons of different configurations across tasks grouped by language types. The value in each pixel block is the percentage rank of the row configuration minus that of column configuration. For example, in word similarity for fusional language, the rank for the row _sms.ww_ is \(0.787\) and \(0.071\) for column _bpe.w-_, and the block value is \(0.716\). The higher the value, the better the performance of the row compared to the column configuration.

 

_ww_, and _sms_ is generally better than _morf_ and _bpe_ in frusional and agglutinative languages (see Figure 4 and 7). The presence of _ww_ is desired in this task as also found by Bojanowski et al. (2017): it enhances the information provided by the segmentation. As discussed before, the best configurations with _bpe_ are always coupled with _ww_, and the worst with _w-ww_ is less important for the more conservative _morf_ where the information stored in _www_ can be fully recovered from the generated subwords. Interestingly, _pp_ and _mp_ do not have positive effects on this task for fusional and introflexive languages, but they seem to resonate well with agglutinative languages, and they are useful for the two other tasks (seen in Figure 6). In general, position embeddings have shown potential benefits in all tasks, where they selectively emphasize or filter subwords according to their positions. _pp_ is extremely useful in entity typing for all languages, because it indicates the root position.

Concerning composition functions, _add_ still remains an extremely robust choice across tasks and languages. Surprisingly, the more sophisticated self-attention composition prevails only on a handful of datasets: compare the results with _add_ vs. _att_ and _mtx_. In fact, the worst configurations mostly use _att_ and _mtx_ (see also Figure 5). In sum, our results suggest that, when unsure, _add_ is by far the most robust choice for the subword composition function. Further, morphotactic tags encoded in subword embeddings (_st_) seem to be only effective combined with self-attention in word similarity and relatedness. These findings call for further investigation in future work, along with the inclusion of finer-grained morphotactic tags into the proposed modeling framework.

Further DiscussionA recurring theme of this study is that subword-informed configurations are largely task- and language-dependent. We can extract multiple examples from the reported results affirming this conjecture. For instance, in fisional and agglutinative languages _mp_ is critical to the model on dependency parsing, while for Hebrew, an introflexive language, _mp_ is among the most detrimental components on the same task. Further, for Turkish word similarity _bpe.ww_ outperforms _sms.ww_: due to affix concatenation in Turkish, _sms_ produces many affixes with only syntactic functions that bring noise to the task. Interestingly, sgns performs well in Hebrew on parsing and word similarity: it shows that it is still difficult for linear segmentation methods to capture nonconcatenative morphology.

Finally, fine-tuning subword-informed representations seems especially beneficial for rare word semantics: our best configuration outperforms ft by \(0.111\) on card, and even surpasses all the state-of-the-art models on the rare word similarity task, as reported by Pilehvar et al. (2018). We hope that our findings on the card dataset will motivate further work on building more accurate representations for rare and unseen words Bhatia et al. (2016); Herbelot and Baroni (2017); Schick and Schutze (2018) by learning more effective and more informed components of subword-informed configurations.

## 5 Conclusion

We have presented a general framework for learning subword-informed word representations which has been used to perform a systematic analysis of 60 different subword-aware configurations for 5 typologically diverse languages across 3 diverse tasks. The large space of presented results has allowed us to analyze the main properties of subword-informed representation learning: we have demonstrated that different components of the framework such as segmentation and composition methods, or the use of position embeddings, have to be carefully tuned to yield improved performance across different tasks and languages. We hope that this study will guide the development of new subword-informed word representation architectures. Code is available at: [https://github.com/cambridgeltl/sw_study](https://github.com/cambridgeltl/sw_study).

## Acknowledgments

This work is supported by the ERC Consolidator Grant LEXICAL (no 648909). We thank the reviewers for their insightful comments, and Roi Reichart for many fruitful discussions.

## References

* Ammar et al. (2016) Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Many languages, one parser. _Transactions of the ACL_, 4:431-444.
* Avraham and Goldberg (2017) Oded Avraham and Yoav Goldberg. 2017. The interplay of semantics and morphology in word embeddings. In _Proceedings of EACL_, pages 422-426.
* Bhatia et al. (2016) Parminder Bhatia, Robert Guthrie, and Jacob Eisenstein. 2016. Morphological priors for probabilisation. _Journal of Statistical Planning Research_, 10:1-16.

 

tic neural word embeddings. In _Proceedings of EMNLP_, pages 490-500.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the ACL_, 5:135-146.
* Chaudhary et al. (2018) Aditi Chaudhary, Chunting Zhou, Lori Levin, Graham Neubig, David R. Mortensen, and Jaime Carbonell. 2018. Adapting word embeddings to new languages with morphological and phonological subword representations. In _Proceedings of EMNLP_, pages 3285-3295.
* Chen and Manning (2014) Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proceedings of EMNLP_, pages 740-750.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* Cotterell et al. (2015) Ryan Cotterell, Thomas Muller, Alexander M. Fraser, and Hinrich Schutze. 2015. Labeled morphological segmentation with semi-Markov models. In _Proceedings of CoNLL_, pages 164-174.
* Cotterrell and Schutze (2015) Ryan Cotterell and Hinrich Schutze. 2015. Morphological word-embeddings. In _Proceedings of NAACL_, pages 1287-1292.
* Cotterell and Schutze (2018) Ryan Cotterell and Hinrich Schutze. 2018. Joint semantic synthesis and morphological analysis of the derived word. _Transactions of the ACL_, 6:33-48.
* Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In _Proceedings of ICLR_.
* Dozat et al. (2017) Timothy Dozat, Peng Qi, and Christopher D. Manning. 2017. Stanford's graph-based neural dependency parser at the CoNLL 2017 shared task. In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_, pages 20-30.
* Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _Journal of Machine Learning Research_, 12:2121-2159.
* word similarity and relatedness. In _Proceedings of COLING_, pages 3819-3836.
* Finkelstein et al. (2002) Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. _ACM Transactions on Information Systems_, 20(1):116-131.
* Gage (1994) Philip Gage. 1994. A new algorithm for data compression. _C Users J._, 12(2):23-38.
* Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. 2017. Convolutional sequence to sequence learning. In _Proceedings of ICML_, pages 1243-1252.
* Gerz et al. (2018) Daniela Gerz, Ivan Vulic, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In _Proceedings of EMNLP_, pages 316-327.
* Goldberg (2017) Yoav Goldberg. 2017. _Neural Network Methods for Natural Language Processing_. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
* Harris (1954) Zellig Harris. 1954. Distributional structure. _Word_, 10(23):146-162.
* Heinzerling and Strube (2018) Benjamin Heinzerling and Michael Strube. 2018. BPEmb: Tokenization-free pre-trained subword embeddings in 275 languages. In _Proceedings of LREC_.
* Herbelot and Baroni (2017) Aurelie Herbelot and Marco Baroni. 2017. High-risk learning: acquiring new word vectors from tiny data. In _Proceedings of EMNLP_, pages 304-309.
* Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. _Computational Linguistics_, 41(4):665-695.
* Jia and Liang (2016) Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In _Proceedings of ACL_, pages 12-22.
* Kiprewasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. _Transaction of the ACL_, 4:313-327.
* Kudo (2018) Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In _Proceedings of ACL_, pages 66-75.
* Lazaridou et al. (2013) Angeliki Lazaridou, Marco Marelli, Roberto Zamarelli, and Marco Baroni. 2013. Compositional-ly derived representations of morphologically complex words in distributional semantics. In _Proceedings of ACL_, pages 1517-1526.
* Leviant and Reichart (2015) Ira Leviant and Roi Reichart. 2015. Judgment language matters: Multilingual vector space models for judgment language aware lexical semantics. _CoRR_.
* Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In _Proceedings of the NIPS_, pages 2177-2185.
* Levy et al. (2015) Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. _Transactions of the ACL_, 3:211-225 

Bofang Li, Aleksandr Drozd, Tao Liu, and Xiaoyong Du. 2018. Subword-level composition functions for learning word embeddings. In _Proceedings of the Second Workshop on Subword/Character LEvel Models_, pages 38-48.
* Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In _Proceedings of ICLR_.
* Ling and Weld (2012) Xiao Ling and Daniel S. Weld. 2012. Fine-grained entity recognition. In _Proceedings of AAAI_, pages 94-100.
* Luong et al. (2013) Thang Luong, Richard Socher, and Christopher Manning. 2013. Better word representations with recursive neural networks for morphology. In _Proceedings of CoNLL_, pages 104-113.
* Ma and Hovy (2017) Xuezhe Ma and Eduard H. Hovy. 2017. Neural probabilistic model for non-projective MST parsing. In _Proceedings of IJCNLP_, pages 59-69.
* Ma et al. (2018)Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stackpointer networks for dependency parsing. In _Proceedings of ACL_, pages 1403-1414.
* Mikolov et al. (2018) Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In _Proceedings of LREC_.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Proceedings of NIPS_, pages 3111-3119.
* Mrksic et al. (2017) Nikola Mrksic, Ivan Vulic, Diarmuid O Seaghdha, Ira Leviant, Roi Reichart, Milica Gasic, Anna Korhonen, and Steve J. Young. 2017. Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints. _Transactions of the ACL_, 5:309-324.
* Nurse et al. (2016) Joakim Nurse, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan T. McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In _Proceedings of LREC_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of EMNLP_, pages 1532-1543.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of NAACL_, pages 2227-2237.
* Pilehvar et al. (2018) Mohammad Taier Pilehvar, Dimitri Kartsaklis, Victor Prokhorov, and Nigel Collier. 2018. Card-660: A reliable evaluation framework for rare word representation models. In _Proceedings of EMNLP_, pages 1391-1401.
* Pinter et al. (2017)Yuval Pinter, Robert Gutthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In _Proceedings of EMNLP_, pages 102-112.
* Qiu et al. (2014) Siyu Qiu, Qing Cui, Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014. Co-learning of word representations and morpheme representations. In _Proceedings of COLING_, pages 141-150.
* Sarawagi and Cohen (2005) Sunita Sarawagi and William W Cohen. 2005. Semi-Markov conditional random fields for information extraction. In _Proceedings of NIPS_, pages 1185-1192.
* Schick and Schutze (2018) Timo Schick and Hinrich Schutze. 2018. Learning semantic representations for novel words: Leveraging both form and context. _CoRR_, abs/1811.03866.
* Sennrich et al. (2016) Rico Sennrich, Barry Hadlow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of ACL_, pages 86-96.
* Shi et al. (2017) Tianze Shi, Liang Huang, and Lillian Lee. 2017. Fast(er) exact decoding and global training for transition-based dependency parsing via a minimal feature set. In _Proceedings of EMNLP_, pages 12-23.
* Smit et al. (2014) Peter Smit, Samir Virpioja, Stig-Arne Grnroos, and Mikko Kurimo. 2014. Morfessor 2.0: Toolkit for statistical morphological segmentation. In _Proceedings of EACL_, pages 21-24.
* Vania and Lopez (2017) Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In _Proceedings of ACL_, pages 2016-2027.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Proceedings of NIPS_, pages 5998-6008.
* Venekoski and Vankka (2017) Viljami Venekoski and Jouko Vankka. 2017. Finnish resources for evaluating language model semantics. In _Proceedings of NoDaLiDa_, pages 231-236.
* Vrandecic and Kotzsch (2014) Denny Vrandecic and Markus Kotzsch. 2014. Wikida: A free collaborative knowledge base. _Communications of the ACM_, 57:78-85.
* Wieting et al. (2016) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Charagram: Embedding words and sentences via character n-grams. In _Proceedings of EMNLP_, pages 1504-1515.
* Yaghoobzadeh and Schutze (2015) Yadollah Yaghoobzadeh and Hinrich Schutze. 2015. Corpus-level fine-grained entity typing using contextual information. In _Proceedings of EMNLP_, pages 715-725 

Jinman Zhao, Sidharth Mudgal, and Yingyu Liang. 2018. Generalizing word embeddings using bag of subwords. In _Proceedings of EMNLP_, pages 601-606.

## Appendix A Supplemental Material

In this supplementary material we present the full results of different configurations of our subword-informed representations and baseline models in three tasks across five languages. For details and notations of different model configurations, please refer to the original paper.

Since fastText (FT) package cannot generate each subword embedding for a given word, we also implemented our own version (our_ft) within the same subword-informed training framework. We trained our_ft on all the five languages with the same training corpus, and kept the same hyperparameters as ft package except that we used 1024 as batch size for faster training. We show that our implementation yields comparable results with the original ft in word similarity and relatedness and parsing across languages. For entity typing, we experimented with the extended hierarchical composition architecture described in the paper on our_ft, i.e., using subword embeddings to compute word representations and updating subword embeddings of our_ft during training, and only use word embeddings as input for sgns and ft and directly update them. Additionally, we also experimented with some of our model configurations with character n-grams as our segmentation methods, but did not observe performance gains. Due to its large parameter space (usually 2-4 times larger than sgns) and the computational limits, we leave it as our future work.

Table 7, 8 and 9 show the results of word similarity and relatedness based on different segmentation methods, and Table 10, 11 and 12 for the results of dependency parsing. Table 13 shows the accuracy on development and test set for entity typing across all five languages. Figure 3 shows the configuration comparisons in languages with available datasets for word relatedness. Figure 4 to 7 show the results on all downstream tasks across languages when each test data point is categorized according to some specific component. Specifically, Figure 4 shows the results when only different segmentation methods are considered, and Figure 5, 6 and 7 show the results on varying composition functions, position embedding types and whether including the word token embedding, respectively

















\begin{table}
\begin{tabular}{l l c c c c c c c} \hline  & dev(en) & test & dev(de) & test & dev(fi) & test & dev(tr) & test \\ \cline{2-9} _sms.w.-p.-add_ & 92.2(uas) & 91.5 & 91.3 & 89.0 & 88.8 & 89.6 & 70.0 & 71.9 \\ _sms.w.-st.add_ & 90.3(las) & 89.6 & 87.7 & 84.2 & 85.5 & 86.1 & 62.1 & 63.7 \\ \cline{2-9} _sms.w.-st.add_ & 92.4 & 91.7 & 91.1 & 88.7 & 89.1 & 89.7 & 71.0 & 72.0 \\ _sms.w.-gt.add_ & 90.5 & 89.7 & 87.6 & 83.8 & 86.0 & 86.6 & 62.9 & 63.8 \\ \cline{2-9} _sms.w.-pp.add_ & 92.2 & 91.9 & 91.4 & 88.7 & 89.3 & 90.5 & 70.9 & 72.7 \\ _sms.w.-mp.add_ & 90.2 & 89.9 & 87.8 & 84.0 & 86.1 & 87.1 & 62.8 & 64.4 \\ \cline{2-9} _sms.w.-mp.add_ & 92.2 & 91.9 & 91.1 & 89.1 & 89.0 & 89.9 & 70.5 & 72.6 \\ _sms.ww.p.-add_ & 90.3 & 90.0 & 87.6 & 84.3 & 85.8 & 86.6 & 62.6 & 64.5 \\ \cline{2-9} _sms.ww.p.-add_ & 92.2 & 92.1 & 91.6 & 89.1 & 89.4 & 90.1 & 71.0 & 71.8 \\ _sms.ww.q._ & 90.2 & 90.0 & 88.1 & 84.3 & 86.4 & 86.9 & 62.8 & 63.6 \\ \cline{2-9} _sms.ww.st.add_ & 92.4 & 91.9 & 91.3 & 89.2 & 89.4 & 89.9 & 71.0 & 72.0 \\ _sms.ww.p.add_ & 90.5 & 89.8 & 87.9 & 84.5 & 86.2 & 86.5 & 63.0 & 64.0 \\ \cline{2-9} _sms.ww.p. add_ & 92.5 & 91.9 & 91.3 & 88.9 & 89.5 & 90.3 & 70.7 & 72.5 \\ _sms.ww.p.add_ & 90.5 & 90.0 & 87.8 & 84.2 & 86.5 & 87.0 & 63.0 & 64.4 \\ \cline{2-9} _sms.ww.mpt.add_ & 92.2 & 91.7 & 91.4 & 89.2 & 89.4 & 90.3 & 71.1 & 72.8 \\ _sms.ww.p.-att_ & 90.2 & 89.6 & 87.7 & 84.4 & 86.3 & 87.1 & 63.5 & 64.7 \\ \cline{2-9} _sms.ww.-gt.att_ & 92.2 & 91.6 & 91.4 & 88.9 & 89.0 & 89.6 & 71.3 & 72.4 \\ _sms.ww.st.att_ & 90.3 & 89.6 & 87.7 & 84.2 & 85.9 & 86.4 & 63.1 & 63.8 \\ \cline{2-9} _sms.ww.st.att_ & 91.4 & 91.2 & 91.1 & 89.0 & 88.8 & 89.4 & 70.0 & 71.9 \\ _sms.ww.pm.att_ & 89.4 & 89.0 & 87.6 & 84.3 & 85.7 & 86.1 & 62.6 & 63.0 \\ \cline{2-9} _sms.ww.mt_ & 92.1 & 91.8 & 91.3 & 89.0 & 89.0 & 89.0 & 70.5 & 72.7 \\ _sms.ww.p.att_ & 90.2 & 89.6 & 87.7 & 84.5 & 85.9 & 86.4 & 63.2 & 64.0 \\ \cline{2-9} _sms.ww.st.att_ & 92.3 & 91.9 & 91.3 & 89.2 & 88.7 & 89.7 & 70.4 & 72.2 \\ _sms.ww.st.att_ & 90.4 & 89.9 & 87.8 & 84.3 & 85.5 & 86.2 & 62.3 & 63.8 \\ \cline{2-9} _sms.ww.pmt_ & 92.2 & 91.8 & 91.5 & 89.0 & 88.5 & 89.3 & 71.0 & 72.2 \\ _sms.ww.pmt_ & 90.3 & 89.8 & 87.9 & 84.4 & 84.9 & 85.8 & 62.7 & 63.7 \\ \cline{2-9} _sms.ww.mp.att_ & 92.3 & 91.9 & 91.5 & 89.1 & 88.6 & 89.5 & 70.7 & 72.5 \\ _sms.ww.p.-ntx_ & 90.1 & 89.8 & 87.9 & 83.7 & 85.9 & 86.2 & 62.5 & 63.5 \\ \cline{2-9} _sms.w.-st.mtx_ & 92.1 & 91.8 & 91.1 & 88.5 & 89.0 & 89.5 & 69.8 & 71.8 \\ _sms.w.-gt.mtx_ & 90.2 & 89.7 & 87.8 & 83.5 & 85.6 & 85.9 & 61.8 & 63.3 \\ \cline{2-9} _sms.w.-pp.mtx_ & 92.3 & 91.7 & 91.2 & 89.1 & 88.9 & 89.6 & 70.7 & 72.0 \\ _sms.ww.mp.mtx_ & 90.4 & 89.7 & 87.6 & 84.2 & 85.9 & 86.3 & 62.6 & 63.1 \\ \cline{2-9} _sms.ww.mp.mtx_ & 92.3 & 92.0 & 91.4 & 89.1 & 89.4 & 90.2 & 70.7 & 72.4 \\ _sms.ww.p.-mx_ & 90.3 & 90.1 & 87.8 & 84.3 & 86.0 & 86.9 & 62.5 & 64.2 \\ \cline{2-9} _sms.ww.-ptr.mix_ & 92.2 & 91.6 & 91.4 & 89.1 & 89.2 & 89.8 & 70.5 & 71.9 \\ _sms.ww.st.mtx_ & 90.4 & 89.6 & 87.9 & 84.3 & 86.0 & 86.4 & 62.5 & 63.6 \\ \cline{2-9} _sms.ww.st.mtx_ & 92.2 & 91.6 & 91.4 & 88.7 & 89.1 & 89.8 & 71.0 & 71.8 \\ _sms.ww.st.mtx_ & 90.4 & 89.7 & 87.8 & 83.9 & 85.8 & 86.5 & 62.7 & 63.3 \\ \cline{2-9} _sms.ww.pp.mtx_ & 92.3 & 92.0 & 91.2 & 89.0 & 88.7 & 89.4 & 70.4 & 71.8 \\ _sms.ww.mp.mtx_ & 90.4 & 89.9 & 87.7 & 84.2 & 85.2 & 85.8 & 62.7 & 63.6 \\ \cline{2-9} _sms.ww.mp.mtx_ & 92.3 & 92.0 & 91.4 & 88.9 & 88.8 & 89.6 & 71.0 & 71.5 \\ _sms.ww.p.-mx_ & 90.4 & 89.9 & 87.8 & 84.0 & 85.2 & 85.9 & 62.7 & 63.3 \\ \cline{2-9} _sms.ww.p.-mx_ & 92.2 & 91.9 & 91.4 & 89.3 & 88.9 & 89.5 & 70.5 & 72.2 \\ _sms.ww.st.mtx_ & 90.4 & 89.8 & 87.9 & 84.4 & 85.6 & 86.2 & 62.5 & 63.5 \\ \cline{2-9} _sms.ww.st.mtx_ & 92.2 & 91.6 & 89.1 & 89.7 & 90.4 & 71.2 & 73.1 \\ _sms.ww.pp.mtx_ & 90.4 & 89.9 & 87.9 & 84.4 & 86.9 & 87.1 & 63.3 & 65.1 \\ \cline{2-9} _sms.





\begin{table}
\begin{tabular}{l l c c c c c c c c c} \hline  & dev(en) & test & dev(de) & test & dev(fi) & test & dev(tr) & test & dev(he) & test \\ \cline{2-13} _morf.w-p.-add_ & 92.3 & 91.7 & 91.1 & 89.0 & 89.0 & 90.0 & 70.7 & 71.8 & 92.5 & 91.0 \\ _morf.w-pp.add_ & 90.4 & 89.8 & 87.7 & 84.2 & 85.8 & 86.6 & 62.4 & 63.3 & 89.6 & 88.3 \\ \cline{2-13} _morf.w-pp.add_ & 92.5 & 91.8 & 91.4 & 89.0 & 89.2 & 89.8 & 70.3 & 71.3 & 92.3 & 91.2 \\ _morf.w-mp.add_ & 90.6 & 89.8 & 87.6 & 84.1 & 86.0 & 86.4 & 62.2 & 63.1 & 89.6 & 88.4 \\ \cline{2-13} _morf.w-mp.add_ & 92.2 & 91.9 & 91.4 & 89.0 & 89.2 & 89.8 & 71.2 & 71.7 & 92.1 & 90.8 \\ _morf.w-mp.add_ & 90.2 & 90.0 & 87.8 & 84.1 & 85.9 & 86.4 & 63.2 & 63.6 & 89.2 & 88.0 \\ \cline{2-13} _morf.ww,p-add_ & 92.2 & 91.8 & 91.4 & 89.0 & 89.4 & 89.6 & 70.8 & 72.2 & 92.3 & 91.2 \\ _morf.ww,p-add_ & 90.3 & 89.7 & 87.8 & 84.1 & 86.2 & 86.2 & 62.9 & 64.2 & 89.5 & 88.5 \\ \cline{2-13} _morf.ww,pp.add_ & 92.3 & 91.7 & 91.3 & 88.9 & 88.8 & 89.9 & 70.6 & 71.9 & 92.2 & 91.0 \\ _morf.ww,pp.add_ & 90.4 & 89.8 & 87.6 & 84.1 & 85.7 & 86.5 & 62.5 & 63.4 & 89.6 & 88.1 \\ \cline{2-13} _morf.ww,pp.add_ & 92.3 & 91.7 & 91.2 & 89.0 & 89.3 & 89.8 & 70.5 & 72.4 & 92.1 & 90.9 \\ _morf.ww,pp.add_ & 90.4 & 89.6 & 87.6 & 84.2 & 86.0 & 86.4 & 62.8 & 64.2 & 89.2 & 88.0 \\ \cline{2-13} _morf.w-p.-att_ & 92.2 & 91.8 & 91.2 & 89.1 & 89.1 & 90.2 & 70.3 & 72.1 & 92.3 & 91.1 \\ _morf.w-pp.att_ & 90.3 & 89.7 & 87.6 & 83.9 & 85.9 & 86.6 & 61.9 & 63.6 & 89.6 & 88.3 \\ \cline{2-13} _morf.w-pp.att_ & 92.3 & 92.0 & 91.5 & 89.3 & 89.1 & 89.7 & 71.0 & 72.2 & 92.6 & 91.2 \\ _morf.w-mp.att_ & 90.3 & 89.9 & 87.9 & 84.6 & 85.9 & 86.4 & 63.1 & 63.6 & 89.7 & 88.4 \\ \cline{2-13} _morf.w-mp.att_ & 92.0 & 91.8 & 91.4 & 89.1 & 89.1 & 89.9 & 70.2 & 71.5 & 92.1 & 90.3 \\ _morf.ww,p-att_ & 90.0 & 89.8 & 87.7 & 84.3 & 85.7 & 86.4 & 62.1 & 62.8 & 89.2 & 87.4 \\ \cline{2-13} _morf.ww,p-att_ & 92.2 & 91.9 & 91.4 & 89.1 & 89.4 & 89.8 & 70.0 & 71.2 & 92.2 & 91.1 \\ _morf.ww,p-att_ & 90.2 & 89.8 & 88.0 & 84.2 & 86.0 & 86.5 & 62.2 & 62.8 & 89.3 & 88.2 \\ \cline{2-13} _morf.ww,pp.att_ & 92.3 & 91.9 & 91.6 & 89.3 & 89.0 & 89.6 & 70.9 & 72.0 & 92.2 & 91.1 \\ _morf.ww,pp.att_ & 90.3 & 89.9 & 88.0 & 84.5 & 85.8 & 86.3 & 62.7 & 63.4 & 89.6 & 88.3 \\ \cline{2-13} _morf.ww,pp.att_ & 92.2 & 91.9 & 91.3 & 89.0 & 88.9 & 90.0 & 70.9 & 71.8 & 92.3 & 91.0 \\ _morf.ww,pp.att_ & 90.4 & 90.0 & 87.9 & 84.1 & 85.8 & 86.7 & 62.8 & 63.4 & 89.4 & 88.0 \\ \cline{2-13} _morf.ww,p-mtx_ & 92.1 & 91.9 & 91.4 & 88.9 & 89.3 & 89.8 & 70.6 & 71.9 & 92.2 & 90.6 \\ _morf.ww,pp.mtx_ & 90.1 & 89.8 & 87.8 & 84.1 & 86.1 & 86.4 & 62.8 & 63.6 & 89.4 & 87.9 \\ \cline{2-13} _morf.ww,pp.mtx_ & 92.4 & 91.8 & 91.3 & 88.8 & 89.3 & 90.0 & 71.0 & 72.2 & 92.3 & 91.3 \\ _morf.w-pp.mtx_ & 90.4 & 89.9 & 87.7 & 84.0 & 86.1 & 86.7 & 63.1 & 63.7 & 89.5 & 88.5 \\ \cline{2-13} _morf.ww,pp.matx_ & 92.2 & 92.1 & 91.4 & 88.8 & 89.2 & 90.3 & 70.2 & 72.2 & 92.2 & 90.5 \\ _morf.ww,pp.matx_ & 90.3 & 90.0 & 87.9 & 83.9 & 85.9 & 86.8 & 62.2 & 63.7 & 89.3 & 87.7 \\ \cline{2-13} _morf.ww,p-mtx_ & 92.2 & 91.8 & 91.3 & 88.9 & 89.3 & 89.6 & 70.1 & 71.4 & 92.5 & 91.4 \\ _morf.ww,pp.mtx_ & 90.3 & 89.8 & 87.9 & 84.0 & 85.9 & 86.1 & 62.1 & 63.3 & 89.6 & 88.4 \\ \cline{2-13} _morf.ww,pp.matx_ & 92.2 & 91.7 & 91.4 & 89.0 & 89.2 & 89.9 & 71.3 & 72.1 & 92.6 & 90.8 \\ _morf.ww,pp.matx_ & 90.3 & 89.6 & 87.9 & 84.1 & 86.0 & 86.7 & 62.9 & 63.8 & 89.7 & 88.0 \\ \cline{2-13} _morf.ww,mp.mtx_ & 92.3 & 91.8 & 91.2 & 89.1 & 89.5 & 89.8 & 71.1 & 72.2 & 92.1 & 90.8 \\ _morf.ww,pp.matx_ & 90.4 & 89.9 & 87.7 & 84.2 & 86.3 & 86.4 & 62.5 & 63.8 & 89.3 & 88.0 \\ \cline{2-13} _sgns_ & 92.3 & 91.9 & 91.4 & 89.3 & 88.9 & 89.5 & 70.5 & 72.2 & 92.4 & 91.5 \\ \cline{2-13} _gns_ & 90.4 & 89.8 & 87.9 & 84.4 & 85.6 & 86.2 & 62.5 & 63.5 & 89.8 & 88.7 \\ \cline{2-13} _ft_ & 92.3 & 92.1 & 91.6 & 89.1 & 89.7 & 90.4 & 71.2 & 7



\begin{table}
\begin{tabular}{l l c c c c c c c c c} \hline  & dev(en) & test & dev(de) & test & dev(fi) & test & dev(tr) & test & dev(he) & test \\ \cline{2-13} _bpe.w–p.-add_ & 92.3 & 91.5 & 91.1 & 89.1 & 89.3 & 89.9 & 70.2 & 72.1 & 91.9 & 91.0 \\ \multirow{4}{*}{_bpe.w–pp.add_} & 90.3 & 89.6 & 87.6 & 84.2 & 85.9 & 86.4 & 62.6 & 63.9 & 89.2 & 88.2 \\ \cline{2-13}  & 92.2 & 91.6 & 91.5 & 89.1 & 89.0 & 90.0 & 71.2 & 71.6 & 92.0 & 90.9 \\ \multirow{4}{*}{_bpe.w–mp.add_} & 90.2 & 89.5 & 87.9 & 84.3 & 85.8 & 86.7 & 63.2 & 63.2 & 89.1 & 88.1 \\ \cline{2-13}  & 91.9 & 91.6 & 91.5 & 89.2 & 89.3 & 89.8 & 70.7 & 71.6 & 92.4 & 91.2 \\ \multirow{4}{*}{_bpe.w-mp.add_} & 90.0 & 89.7 & 87.9 & 84.3 & 86.1 & 86.3 & 62.7 & 63.4 & 89.7 & 88.2 \\ \cline{2-13}  & 92.1 & 91.8 & 91.3 & 89.2 & 89.5 & 89.8 & 70.2 & 71.4 & 92.5 & 91.1 \\ \multirow{4}{*}{_bpe.ww.p-add_} & 90.2 & 89.9 & 87.7 & 84.4 & 86.2 & 86.5 & 62.1 & 63.2 & 89.5 & 88.2 \\ \cline{2-13}  & 92.3 & 91.6 & 91.2 & 89.6 & 89.2 & 90.2 & 70.8 & 72.3 & 92.3 & 91.2 \\ \multirow{4}{*}{_bpe.ww.pp.add_} & 90.4 & 89.6 & 87.7 & 84.7 & 86.2 & 87.0 & 62.9 & 64.2 & 89.4 & 88.4 \\ \cline{2-13}  & 92.3 & 91.9 & 91.6 & 89.2 & 89.9 & 90.7 & 71.2 & 72.3 & 92.2 & 91.0 \\ \multirow{4}{*}{_bpe.ww.mp.add_} & 90.3 & 89.9 & 88.1 & 84.5 & 86.9 & 87.4 & 63.1 & 64.0 & 89.4 & 88.2 \\ \cline{2-13}  & 92.3 & 91.3 & 91.4 & 89.1 & 88.8 & 89.4 & 70.4 & 71.9 & 92.0 & 90.9 \\ \multirow{4}{*}{_bpe.w–p.att_} & 90.3 & 89.4 & 87.9 & 84.2 & 85.3 & 85.7 & 62.3 & 63.0 & 89.1 & 87.9 \\ \cline{2-13}  & 92.1 & 91.6 & 91.4 & 88.6 & 88.7 & 89.5 & 70.9 & 71.5 & 91.9 & 91.0 \\ \multirow{4}{*}{_bpe.w–mp.att_} & 90.2 & 89.7 & 87.6 & 83.7 & 85.4 & 86.0 & 62.5 & 63.3 & 89.0 & 88.3 \\ \cline{2-13}  & 92.1 & 91.5 & 91.5 & 89.3 & 89.1 & 89.9 & 70.5 & 72.4 & 92.2 & 90.8 \\ \multirow{4}{*}{_bpe.ww.pm._} & 90.2 & 89.6 & 87.9 & 84.4 & 85.8 & 86.3 & 62.5 & 63.9 & 89.5 & 88.0 \\ \cline{2-13}  & 92.3 & 92.0 & 91.4 & 88.6 & 89.1 & 90.1 & 70.8 & 71.3 & 92.0 & 90.8 \\ \multirow{4}{*}{_bpe.ww:p–att_} & 90.4 & 89.9 & 87.9 & 83.8 & 85.8 & 86.8 & 62.4 & 63.1 & 89.3 & 88.0 \\ \cline{2-13}  & 92.3 & 91.8 & 91.4 & 89.2 & 89.2 & 90.0 & 71.0 & 72.1 & 92.1 & 91.1 \\ \multirow{4}{*}{_bpe.ww.pp.att_} & 90.5 & 89.8 & 87.7 & 84.6 & 85.9 & 86.4 & 63.2 & 64.0 & 89.5 & 88.2 \\ \cline{2-13}  & 92.5 & 92.0 & 91.5 & 89.3 & 89.5 & 90.3 & 70.6 & 71.7 & 91.9 & 90.9 \\ \multirow{4}{*}{_bpe.ww.mp.att_} & 90.6 & 89.9 & 88.1 & 83.9 & 86.4 & 86.9 & 62.6 & 63.4 & 89.1 & 88.0 \\ \cline{2-13}  & 91.9 & 91.7 & 91.1 & 88.9 & 88.6 & 89.5 & 69.9 & 71.5 & 91.5 & 91.2 \\ \multirow{4}{*}{_bpe.w–p.mtx_} & 90.1 & 89.7 & 87.5 & 83.9 & 85.2 & 85.9 & 61.9 & 63.1 & 88.7 & 88.3 \\ \cline{2-13}  & 92.1 & 91.7 & 91.1 & 88.8 & 89.0 & 89.5 & 70.7 & 71.8 & 92.0 & 91.0 \\ \multirow{4}{*}{_bpe.w–pp.mtx_} & 90.2 & 89.7 & 87.5 & 84.0 & 85.6 & 86.0 & 62.8 & 63.3 & 89.2 & 88.2 \\ \cline{2-13}  & 92.1 & 91.6 & 91.1 & 89.1 & 89.0 & 89.9 & 70.3 & 72.0 & 92.2 & 90.6 \\ \multirow{4}{*}{_bpe.w-mp.mtx_} & 90.2 & 89.5 & 87.6 & 84.1 & 85.7 & 86.3 & 62.6 & 63.6 & 89.3 & 87.7 \\ \cline{2-13}  & 92.3 & 92.1 & 91.6 & 89.0 & 89.3 & 90.0 & 71.7 & 72.3 & 91.9 & 90.5 \\ \multirow{4}{*}{_bpe.ww.p-mtx_} & 90.4 & 90.0 & 88.0 & 84.2 & 86.0 & 86.7 & 63.4 & 63.7 & 88.9 & 87.7 \\ \cline{2-13}  & 92.4 & 91.9 & 91.3 & 89.1 & 89.3 & 89.7 & 69.8 & 72.2 & 92.0 & 90.7 \\ \multirow{4}{*}{_bpe.ww.pp.mtx_} & 90.4 & 89.8 & 87.7 & 84.2 & 85.9 & 86.2 & 62.0 & 63.8 & 89.2 & 87.9 \\ \cline{2-13}  & 92.3 & 91.7 & 91.2 & 89.4 & 89.8 & 90.2 & 71.2 & 72.3 & 92.2 & 91.0 \\ \multirow{4}{*}{_bpe.ww.mp.mtx_} & 90.4 & 89.6 & 87.7 & 84.7 & 86.7 & 86.8 & 63.1 & 64.2 & 89.4 & 88.1 \\ \cline{2-13}  & 92.3 & 91.9 & 91.4 & 89.3 & 88.9 & 89.5 & 70.5 & 72.2 & 92.4 & 91.5 \\ \multirow{4}{*}{sgns} & 90.4 & 89.8 & 87.9 & 84.4 & 85.6 & 86.2 & 62.5 & 63.5 & 89.8 & 88.7 \\ \cline{2-13}  & 92.3 & 92.1 & 91.6 & 89.1 & 89.7 & 90.4 & 71.2 & 73.1 & 9



\begin{table}
\begin{tabular}{l l c c c c c c c c}  & dev(en) & test & dev(de) & test & dev(fi) & test & dev(tr) & test & dev(he) \\ \hline _sms.w-p.-add_ & 52.88 & 52.57 & 51.09 & 50.95 & 55.15 & 55.08 & 53.87 & 53.52 & \\ _sms.w-st.add_ & 52.68 & 52.7 & 51.2 & 51.23 & 55.38 & 55.13 & 54.03 & 53.68 & \\ _sms.w-pp.add_ & 53.19 & 52.94 & 50.48 & 50.83 & 55.44 & 55.30 & 54.05 & 53.88 & \\ _sms.w-mp.add_ & 52.87 & 52.76 & 51.11 & 51.18 & 55.18 & 55.00 & 54.24 & 53.96 & \\ _sms.uw-pd.adj_ & 53.56 & 53.69 & 51.73 & 51.77 & 55.92 & 55.84 & 55.01 & 54.76 & \\ _sms.ww.std.add_ & 54.21 & 54.26 & 52.53 & 52.63 & 56.21 & 55.95 & 55.96 & 55.43 & \\ _sms.www.pp.add_ & 54.51 & 54.66 & 52.98 & 53.04 & 56.06 & 56.00 & 55.92 & 55.75 & \\ _sms.ww.pmd_ & 54.27 & 54.43 & 52.77 & 52.88 & 56.04 & 55.71 & 56.38 & 55.99 & \\ _sms.w-p.att_ & 52.81 & 52.7 & 50.7 & 50.8 & 54.85 & 54.67 & 53.62 & 53.45 & \\ _sms.ws-st.att_ & 52.57 & 52.64 & 51.64 & 51.65 & 54.87 & 54.69 & 53.27 & 53.14 & \\ _sms.wp-pt.att_ & 52.94 & 52.66 & 52.1 & 52.5 & 54.91 & 54.83 & 53.39 & 53.21 & \\ _sms.w-np.att_ & 52.99 & 52.91 & 51.7 & 51.9 & 55.02 & 55.03 & 54.06 & 53.72 & \\ _sms.tw.p-att_ & 53.68 & 53.67 & 51.48 & 51.69 & 55.59 & 55.73 & 55.17 & 54.92 & \\ _sms.ww.st.att_ & 53.55 & 53.66 & 51.64 & 51.65 & 55.27 & 55.35 & 53.84 & 53.26 & \\ _sms.ww.pn.att_ & 53.14 & 53.08 & 52.1 & 52.5 & 54.66 & 54.52 & 54.95 & 54.77 & \\ _sms.ww.app.att_ & 53.82 & 53.79 & 51.67 & 51.89 & 54.02 & 54.04 & 54.1 & 53.76 & \\ _sms.w-p.,mtk_ & 52.7 & 52.67 & 50.29 & 50.52 & 54.61 & 54.51 & 53.44 & 53.24 & \\ _sms.w-sn.tmx_ & 52.75 & 52.61 & 50.42 & 50.85 & 54.15 & 54.09 & 53.44 & 52.84 & \\ _sms.w-pp.mtx_ & 52.89 & 52.87 & 51.07 & 51.38 & 54.58 & 54.70 & 53.42 & 53.16 & \\ _sms.wh-np.mtx_ & 53.03 & 52.89 & 50.52 & 50.95 & 55.19 & 54.90 & 54.23 & 53.68 & \\ _sms.ww.pe-mtx_ & 53.75 & 53.81 & 51.47 & 51.48 & 55.34 & 55.37 & 54.51 & 54.36 & \\ _sms.ww.str.intx_ & 53.75 & 53.64 & 51.24 & 51.51 & 55.4 & 55.36 & 54.97 & 54.72 & \\ _sms.we.pp.mk_ & 53.80 & 53.96 & 51.92 & 52.1 & 54.57 & 54.54 & 54.94 & 54.57 & \\ _sms.www.imp.mtx_ & 54.23 & 54.2 & 52 & 52.2 & 55.02 & 54.84 & 55.8 & 55.45 & \\ _morf w.-p.add_ & 54.86 & 55.35 & 53.41 & 53.28 & 57.01 & 56.94 & 55.92 & 55.8 & 60.37 & 59.67 \\ _morf w.-pp.add_ & 55.55 & 55.57 & 54.02 & 54.06 & 57.69 & 57.41 & 56.79 & 56.31 & 60.61 & 59.99 \\ _morf w.-np.add_ & 55.01 & 55.18 & 53.5 & 53.35 & 57.09 & 56.82 & 55.26 & 54.7 & 60.1 & 59.24 \\ _morf w.-p.add_ & 54.94 & 55.09 & 53.27 & 53.02 & 56.61 & 56.55 & 55.5 & 54.9 & 60.23 & 59.65 \\ _morf w.wp.add_ & 55.20 & 55.60 & 53.83 & 53.82 & 57.32 & 57.2 & 56.41 & 55.92 & 60.57 & 60.09 \\ _morf w.mp.add_ & 54.63 & 54.67 & 53.46 & 53.4 & 56.81 & 56.73 & 54.42 & 53.88 & 59.67 & 58.87 \\ _morf w.-p.ant_ & 54.84 & 55.02 & 52.98 & 52.91 & 56.93 & 56.83 & 55.86 & 55.52 & 60.3 & 59.78 \\ _morf w.-pp.att_ & 55.28 & 55.68 & 53.92 & 54.01 & 57.36 & 57.24 & 56.62 & 56.13 & 60.83 & 60.34 \\ _morf w.-mpi.att_ & 55.4 & 55.3 & 53.72 & 53.7 & 56.85 & 56.93 & 55.24 & 54.6 & 59.89 & 59.38 \\ _morf w.-pw.att_ & 54.94 & 54.87 & 53.07 & 53.12 & 56.7 & 56.36 & 55.55 & 55.07 & 60.31 & 59.73 \\ _morf w.wp.opt.att_ & 55.26 & 55.35 & 53.83 & 53.92 & 57.39 & 57.15 & 56.39 & 55.99 & 60.5 & 59.94 \\ _morf w.mp.att_ & 55.16 & 55.27 & 53.22 & 53.26 & 56.84 & 56.67 & 54.89 & 54.32 & 59.59 & 59.04 \\ _morf w.-pv.mtx_ & 54.97 & 55.06 & 52.39 & 52.42 & 56.76 & 56.88 & 55.93 & 55.39 & 60.26 & 59.66 \\ _morf w.-pp.mtx_ & 55.18 & 55.4 & 53.95 & 53.92 & 57.49 & 57.38 & 56.62 & 56.14 & 60.65 & 60.02 \\ _morf w.-mp.mtx_ & 55.18 & 55.25 & 52.92 & 53.01 & 56.97 & 56.69 & 55.02 & 54.48 & 60.07 & 59.36 \\ _morf w.-pv.-mv.



* [15] M. C. Bader, A. K. Goyal, and R. J. Brox. The role of the agent's attention mechanism in social networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 237-240, 2016.
* [16] S. D. Lee, Y. Zhang, and L. Zisserman. An efficient method for learning multiple images using a deep neural network. In _Advances in Neural Information Processing Systems (NIPS)_, pages 89-92, 2016.
* [17] T. Lin, X. Liu, and H. Yu. Learning multiple image representations by jointly optimizing the latent variable. In _IEEE International Conference on Image Processing (ICIP)_, pages 103-104, 2016.

[MISSING_PAGE_POST]

 

Figure 6: Results with different position embeddings.

Figure 7: Results w/o word token embedding.