Venue: NAACL 2018

# Post-Specialisation: Retrofitting Vectors of Words

Unseen in Lexical Resources

Ivan Vulic\({}^{1}\), Goran Glavas\({}^{2}\), Nikola Mrksic\({}^{3}\), Anna Korhonen\({}^{1}\)

\({}^{1}\) Language Technology Lab, University of Cambridge

\({}^{2}\) Data and Web Science Group, University of Mannheim

\({}^{3}\) PolyAI {iv250,alk23}@cam.ac.uk goran@informatik.uni-mannheim.de nikola@poly-ai.com

###### Abstract

Word vector _specialisation_ (also known as _retrofitting_) is a portable, light-weight approach to fine-tuning arbitrary distributional word vector spaces by _injecting_ external knowledge from rich lexical resources such as WordNet. By design, these _post-processing_ methods only update the vectors of words occurring in external lexicons, leaving the representations of all _unseen_ words intact. In this paper, we show that constraint-driven vector space specialisation can be extended to unseen words. We propose a novel _post-specialisation_ method that: **a)** preserves the useful linguistic knowledge for seen words; while **b)** propagating this external signal to unseen words in order to improve their vector representations as well. Our post-specialisation approach explicits a non-linear specialisation function in the form of a deep neural network by learning to predict specialised vectors from their original distributional counterparts. The learned function is then used to specialise vectors of unseen words. This approach, applicable to any post-processing model, yields considerable gains over the initial specialisation models both in intrinsic word similarity tasks, and in two downstream tasks: _dialogue state tracking_ and _lexical text simplification_. The positive effects persist across three languages, demonstrating the importance of specialising the full vocabulary of distributional word vector spaces.

## 1 Introduction

Word representation learning is a key research area in current Natural Language Processing (NLP), with its usefulness demonstrated across a range of tasks Collobert et al. (2011); Chen and Manning (2014); Melamud et al. (2016). The standard techniques for inducing distributed word representations are grounded in the distributional hypothesis Harris (1954): they rely on co-occurrence information in large textual corpora Mikolov et al. (2013); Pennington et al. (2014); Levy and Goldberg (2014); Levy et al. (2015); Bojanowski et al. (2017). As a result, these models tend to coalesce the notions of semantic similarity and (broader) conceptual relatedness, and cannot accurately distinguish antonyms Hill et al. (2015); Schwartz et al. (2015). Recently, we have witnessed a rise of interest in representation models that move beyond stand-alone unsupervised learning: they leverage external knowledge in human- and automatically-constructed lexical resources to enrich the semantic content of distributional word vectors, in a process termed _semantic specialisation_.

This is often done as a post-processing (sometimes referred to as _retrofitting_) step: input word vectors are _fine-tuned_ to satisfy linguistic constraints extracted from lexical resources such as WordNet or BabelNet Faruqui et al. (2015); Mrksic et al. (2017). The use of external curated knowledge yields improved word vectors for the benefit of downstream applications Faruqui (2016). At the same time, this specialisation of the distributional space distinguishes between true similarity and relatedness, and supports language understanding tasks Kiela et al. (2015); Mrksic et al. (2017).

While there is consensus regarding their benefits and ease of use, one property of the post-processing specialisation methods slips under the radar: most existing post-processors update word embeddings only for words which are present (i.e., _seen_) in the external constraints, while vectors of all other (i.e., _unseen_) words remain unaffected. In this work, we propose a new approach that extends the specialisation framework to unseen words, relying on the transformation of the vector (sub)space of seen words. Our intuition is that the process of fine-tuning seen words provides implicit information on how to leverage the external knowledge to unseen words. The method should preserve the already injected knowledge for seen words, simultaneously

propagating the external signal to unseen words in order to improve their vectors.

The proposed _post-specialisation_ method can be seen as a two-step process, illustrated in Fig. 1a: **1)** We use a state-of-the-art specialisation model to transform the subspace of seen words from the input distributional space into the specialised subspace; **2)** We learn a mapping function based on the transformation of the "seen subspace", and then apply it to the distributional subspace of unseen words. We allow the proposed post-specialisation model to learn from large external linguistic resources by implementing the mapping as a deep feed-forward neural network with non-linear activations. This allows the model to learn the generalisation of the fine-tuning steps taken by the initial specialisation model, itself based on a very large number (e.g., hundreds of thousands) of external linguistic constraints.

As indicated by the results on word similarity and two downstream tasks (dialogue state tracking and lexical text simplification) our post-specialisation method consistently outperforms state-of-the-art methods which specialise seen words only. We report improvements using three distinct input vector spaces for English and for three test languages (English, German, Italian), verifying the robustness of our approach.

## 2 Related Work and Motivation

Vector Space SpecialisationA standard approach to incorporating external and background knowledge into word vector spaces is to pull the representations of similar words closer together and to push words in undesirable relations (e.g., antonyms) away from each other. Some models integrate such constraints into the training procedure and _jointly_ optimize distributional and non-distributional objectives: they modify the prior or the regularisation Yu and Dredze (2014); Xu et al. (2014); Bian et al. (2014); Kiela et al. (2015), or use a variant of the SGNS-style objective Liu et al. (2015); Ono et al. (2015); Osborne et al. (2016); Nguyen et al. (2017). In theory, word embeddings obtained by these joint models could be as good as representations produced by models which finetune input vector space. However, their performance falls behind that of fine-tuning methods Wieting et al. (2015). Another disadvantage is that their architecture is tied to a specific underlying model (typically word2vec models).

In contrast, fine-tuning models inject external knowledge from available lexical resources (e.g., WordNet, PPDB) into pre-trained word vectors as a _post-processing step_Faruqui et al. (2015); Rothe and Schutze (2015); Wieting et al. (2015); Nguyen et al. (2016); Mrksic et al. (2016); Cotterell et al. (2016); Mrksic et al. (2017). Such post-processing models are popular because they offer a portable, flexible, and light-weight approach to incorporating external knowledge into _arbitrary_ vector spaces, yielding state-of-the-art results on language understanding tasks Faruqui et al. (2015); Mrksic et al. (2016); Kim et al. (2016); Vulic et al. (2017b).

Existing post-processing models, however, suffer from a major limitation. Their _modus operandi_ is to enrich the distributional information with external knowledge only if such knowledge is present in a lexical resource. This means that they update and improve only representations of words actually seen in external resources. Because such words constitute only a fraction of the whole vocabulary (see Sect. 4), most words, unseen in the constraints, retain their original vectors. The main goal of this work is to address this shortcoming by specialising _all_ words from the initial distributional space.

## 3 Methodology: Post-Specialisation

Our starting point is the state-of-the-art specialisation model attract-repel (ar) Mrksic et al. (2017), outlined in Sect. 3.1. We opt for the ar model due to its strong performance and ease of use, but we note that the proposed post-specialisation approach for specialising unseen words, described in Sect. 3.2, is applicable to any post-processor, as empirically validated in Sect. 5.

### Initial Specialisation Model: ar

Let \({\cal V}_{s}\) be the vocabulary, \(A\) the set of synonymous attract word pairs (e.g., _rich_ and _wealthhy_), and \(R\) the set of anonymous repel word pairs (e.g., _increase_ and _decrease_). The attract-repel procedure operates over mini-batches of such pairs \({\cal B}_{A}\) and \({\cal B}_{R}\). Let each word pair \((x_{l},x_{r})\) in these sets correspond to a vector pair \(({\bf x}_{l},{\bf x}_{r})\). A mini-batch of \(b_{att}\) attract word pairs is given by \({\cal B}_{A}=[({\bf x}^{1}_{l},{\bf x}^{1}_{r}),\ldots,({\bf x}^{k_{1}}_{l},{ \bf x}^{k_{1}}_{r})]\) (analogously for \({\cal B}_{R}\), which consists of \(b_{rep}\) pairs).

Next, the sets of negative examples \(T_{A}=[({\bf t}^{1}_{l},{\bf t}^{1}_{r}),\ldots,({\bf t}^{k_{1}}_{l},{\bf t} ^{k_{1}}_{r})]\) and \(T_{R}=[({\bf t}^{1}_{l},{\bf t}^{1}_{r}),\ldots,({\bf t}^{k_{2}}_{l},{\bf t} ^{k_{2}}_{r})]\) are defined as pairs of _negative examples_ for each \(A\) and 

pair in mini-batches \({\cal B}_{A}\) and \({\cal B}_{R}\). These negative examples are chosen from the word vectors present in \({\cal B}_{A}\) or \({\cal B}_{R}\) so that, for each \(A\) pair \(({\bf x}_{l},{\bf x}_{r})\), the negative example pair \(({\bf t}_{l},{\bf t}_{r})\) is chosen so that \({\bf t}_{l}\) is the vector closest (in terms of cosine distance) to \({\bf x}_{l}\) and \({\bf t}_{r}\) is closest to \({\bf x}_{r}\).1 The negatives are used **1)** to force \(A\) pairs to be closer to each other than to their respective negative examples; and **2)** to force \(R\) pairs to be further away from each other than from their negative examples. The first term of the cost function pulls \(A\) pairs together:

Footnote 1: Similarly, for each \(R\) pair \(({\bf x}_{l},{\bf x}_{r})\), the negative pair \(({\bf t}_{i},{\bf t}_{r})\) is chosen from the in-batch vectors so that \({\bf t}_{l}\) is the vector furthest away from \({\bf x}_{l}\) and \({\bf t}_{r}\) is furthest from \({\bf x}_{r}\). All vectors are unit length (re)normalised after each epoch.

\[Att({\cal B}_{A},T_{A})=\sum_{i=1}^{b_{att}} \big{[}\tau\left(\delta_{att}+{\bf x}_{l}^{i}|{\bf t}_{i}^{i}-{\bf x }_{l}^{i}{\bf x}_{r}^{i}\right)\] \[+ \tau\left(\delta_{att}+{\bf x}_{r}^{i}{\bf t}_{r}^{i}-{\bf x}_{l} ^{i}{\bf x}_{r}^{i}\right)\big{]} \tag{1}\]

where \(\tau(z)=\max(0,z)\) is the standard rectifier function Nair and Hinton (2010) and \(\delta_{att}\) is the attract margin: it determines how much closer these vectors should be to each other than to their respective negative examples. The second, repel term in the cost function is analogous: it pushes \(R\) word pairs away from each other by the margin \(\delta_{rep}\).

Finally, in addition to the \(A\) and \(R\) terms, a regularisation term is used to preserve the semantic content originally present in the distributional vector space, as long as this information does not contradict the injected external knowledge. Let \({\mathcal{V}}({\cal B})\) be the set of all word vectors present in a mini-batch, the distributional regularisation term is then:

\[Reg({\cal B}_{A},{\cal B}_{R})=\sum_{{\bf x}_{i}\in{\mathcal{V}}({\cal B}_{A} \cup{\cal B}_{R})}\lambda_{reg}\left\lVert\widehat{{\bf x}_{i}}-{\bf x}_{i} \right\rVert_{2} \tag{2}\]

where \(\lambda_{reg}\) is the \(L_{2}\)-regularisation constant and \(\widehat{{\bf x}_{i}}\) denotes the original (distributional) word vector for word \(x_{i}\). The full attract-repel cost function is finally constructed as the sum of all three terms.

### Specialisation of Unseen Words

Problem FormulationThe goal is to learn a _global transformation function_ that generalises the perturbations of the initial vector space made by attract-repel (or any other specialisation procedure), as conditioned on the external constraints. The learned function propagates the signal coded in the input constraints to all the words unseen during the specialisation process. We seek a regression function \(f:\mathbb{R}^{dim}\rightarrow\mathbb{R}^{dim}\), where \(dim\) is the vector space dimensionality. It maps word vectors from the initial vector space \({\mathbf{X}}\) to the specialised target space \({\mathbf{X}}^{\prime}\). Let \(\widehat{{\mathbf{X}}^{\prime}}=f({\mathbf{X}})\) refer to the predicted mapping of the vector space, while the mapping of a single word vector is denoted \(\widehat{{\mathbf{x}}_{i}}=f({\bf x}_{i})\).

An input distributional vector space \({\mathbf{X}}_{d}\) represents words from a vocabulary \({\mathcal{V}}_{d}\). \({\mathcal{V}}_{d}\) may be divided into two vocabulary subsets: \({\mathcal{V}}_{d}={\mathcal{V}}_{s}\cup{\mathcal{V}}_{u}\), \({\mathcal{V}}_{s}\cap{\mathcal{V}}_{u}=\emptyset\), with the accompanying vector subspaces \({\mathbf{X}}_{d}={\mathbf{X}}_{s}\sqcup{\mathbf{X}}_{u}\). \({\mathcal{V}}_{s}\) refers to the vocabulary of seen words: those that appear in the external linguistic constraints and have their embeddings changed in the specialisation process. \({\mathcal{V}}_{u}\) denotes the vocabulary of unseen words: those not present in the constraints and whose embeddings are unaffected by the specialisation procedure.

The ar specialisation process transforms only the subspace \({\mathbf{X}}_{s}\) into the specialised subspace \({\mathbf{X}}_{s}^{\prime}\). All words \(x_{i}\in{\mathcal{V}}_{s}\) may now be used as training examples for learning the explicit mapping function \(f\) from \({\mathbf{X}}_{s}\) into \({\mathbf{X}}_{s}^{\prime}\). If \(N=|{\mathcal{V}}_{s}|\), we in fact rely on \(N\) training pairs: \(({\bf x}_{i},{\bf x}_{i}^{\prime})=\{{\bf x}_{i}\in{\mathbf{X}}_{s},{\bf x}_{ i}^{\prime}\in{\mathbf{X}}_{s}^{\prime}\}\). Function \(f\) can then be applied to unseen words \(x\in{\mathcal{V}}_{u}\) to yield the specialised subspace \(\widehat{{\mathbf{X}}_{u}^{\prime}}=f({\mathbf{X}}_{u})\). The specialised space containing _all_ words is then \({\mathbf{X}}_{f}={\mathbf{X}}_{s}^{\prime}\cup\widehat{{\mathbf{X}}_{u}^{\prime}}\). The complete high-level post-specialisation procedure is outlined in Fig. 0(a).

Note that another variant of the approach could obtain \({\mathbf{X}}_{f}\) as \({\mathbf{X}}_{f}=f({\mathbf{X}}_{d})\), that is, the entire distributional space is transformed by \(f\). However, this variant seems counter-intuitive as it forgets the actual output of the initial specialisation procedure and replaces word vectors from \({\mathbf{X}}_{s}^{\prime}\) with their approximations, i.e., \(f\)-mapped vectors.2

Footnote 2: We have empirically confirmed the intuition that the first variant is superior to this alternative. We do not report the actual quantitative comparison for brevity.

Objective FunctionsAs mentioned, the \(N\)_seen_ words \(x_{i}\in{\mathcal{V}}_{s}\) in fact serve as our "pseudotranslation" pairs supporting the learning of a cross-space mapping function. In practice, in its high-level formulation, our mapping problem is equivalent to those encountered in the literature on cross-lingual word embeddings where the goal is to learn a shared cross-lingual space given monolingual vector spaces in two languages and \(N_{1}\) translation pairs Mikolov et al. (2013); Lazaridou et al. (2015); Vulic and Korhonen (2016); Artetxe et al. (2016, 2017); Conneau et al. (2017); Ruder et al. (2017). In our setup, the standard objective based 

least squares may be formulated as follows:

\[f_{\text{\tiny{RMSE}}}=\text{arg min}||f(\mathbf{X}_{s})-\mathbf{X}^{\prime}_{ s}||_{F}^{2} \tag{3}\]

where \(||\cdot||_{F}^{2}\) denotes the squared Frobenius norm. In the most common form \(f(\mathbf{X}_{s})\) is simply a linear map/matrix \(\mathbf{W}_{f}\in\mathbb{R}^{dim\times dim}\)Mikolov et al. (2013a) as follows: \(f(\mathbf{X})=\mathbf{W}_{f}\mathbf{X}\).

After learning \(f\) based on the \(\mathbf{X}_{s}\rightarrow\mathbf{X}^{\prime}_{s}\) transformation, one can simply apply \(f\) to unseen words: \(\widehat{\mathbf{X}}^{\prime}_{u}=f(\mathbf{X}_{u})\). This linear mapping model, termed linear-mse, has an analytical solution Artetxe et al. (2016), and has been proven to work well with cross-lingual embeddings. However, given that the specialisation model injects hundreds of thousands (or even millions) of linguistic constraints into the distributional space (see later in Sect. 4), we suspect that the assumption of linearity is too limiting and does not fully hold in this particular setup.

Using the same \(L_{2}\)-penalized least squares objective, we can thus replace the linear map with a _nonlinear_ function \(f:\mathbb{R}^{dim}\to\mathbb{R}^{dim}\). The non-linear mapping, illustrated by Fig. 0(b), is implemented as a deep feed-forward fully-connected neural network (DFFN) with \(H\) hidden layers and non-linear activations. This variant is called nonlinear-mse.

Another variant objective is the contrastive margin-based ranking loss with negative sampling (mm) similar to the original attract-repel objective, used in other applications in prior work (e.g., for cross-modal mapping) Weston et al. (2011); Frome et al. (2013); Lazaridou et al. (2015); Kummerfeld et al. (2015). Let \(\widehat{\mathbf{x}^{\prime}_{i}}=f(\mathbf{x}_{i})\) denote the predicted vector for the word \(x_{i}\in\mathcal{V}_{s}\), and let \(\mathbf{x^{\prime}}_{i}\) refer to the "true" vector of \(x_{i}\) in the specialised space \(\mathbf{X}^{\prime}_{s}\) after the ar specialisation procedure. The mm loss is then defined as follows:

\[J_{\text{\tiny{MM}}}=\sum_{i=1}^{N}\sum_{j\neq i}^{k}\tau\Big{(}\delta_{mm}- cos(\widehat{\mathbf{x}^{\prime}_{i}},\mathbf{x^{\prime}}_{i})+cos( \widehat{\mathbf{x}^{\prime}_{i}},\mathbf{x^{\prime}}_{j})\Big{)}\]

where \(cos\) is the cosine similarity measure, \(\delta_{mm}\) is the margin, and \(k\) is the number of negative samples. The objective tries to learn the mapping \(f\) so that each predicted vector \(\widehat{\mathbf{x}^{\prime}_{i}}\) is by the specified margin \(\delta_{mm}\) closer to the correct target vector \(\mathbf{x^{\prime}}_{i}\) than to any other of \(k\) target vectors \(\mathbf{x^{\prime}}_{j}\) serving as negative examples.3 Function \(f\) can again be either a simple linear map (linear-mm), or implemented as a DFFN (nonlinear-mm, see Fig. 0(b)).

Figure 1: **(a) High-level illustration of the post-specialisation approach: the subspace \(\mathbf{X}_{s}\) of the initial distributional vector space \(\mathbf{X}_{d}=\mathbf{X}_{s}\cup\mathbf{X}_{u}\) is first specialised/fine-tuned by the attract-repel specialization model (or any other post-processing model) to obtain the transformed subspace \(\mathbf{X}^{\prime}_{s}\). The words present (i.e., _seen_) in the input set of linguistic constraints are now assigned different representations in \(\mathbf{X}_{s}\) (the original distributional vector) and \(\mathbf{X}^{\prime}_{s}\) (the specialised vector): they are therefore used as training examples to learn a non-linear cross-space mapping function. This function is then applied to all word vectors \(\mathbf{x}_{i}\in\mathbf{X}_{u}\) representing words _unseen_ in the constraints to yield a specialised subspace \(\widetilde{\mathbf{X}}^{\prime}_{u}\). The final space is \(\mathbf{X}_{f}=\mathbf{X}^{\prime}_{s}\cup\widetilde{\mathbf{X}}^{\prime}_{ u}\), and it contains transformed representations for _all_ words from the initial space \(\mathbf{X}_{d}\). **(b)** The actual implementation of the non-linear regression function which maps from \(\mathbf{X}_{u}\) to \(\widetilde{\mathbf{X}}^{\prime}_{u}\): a deep feed-forward fully-connected neural net with non-linearities and \(H\) hidden layers.

 



## 4 Experimental Setup

**Starting Word Embeddings (X\({}_{d}=\) X\({}_{s}\cup\) X\({}_{u}\))** To test the robustness of our approach, we experiment with three well-known, publicly available collections of English word vectors: **1)** Skip-Gram with Negative Sampling (gns-bow2) Mikolov et al. (2013b) trained on the Polyglot Wikipedia Alfou et al. (2013) by Levy and Goldberg (2014) using bag-of-words windows of size 2; **2)** glove Common Crawl Pennington et al. (2014); and **3)** fastTextBojanowski et al. (2017), a SGNS variant which builds word vectors as the sum of their constituent character n-gram vectors. All word embeddings are \(300\)-dimensional.4

Footnote 4: For further details regarding the architectures and training setup of the used vector collections, we refer the reader to the original papers. Additional experiments with other word vectors, e.g., with context2vecMelamud et al. (2016a) (which uses bidirectional LSTMs Hochreiter and Schmidhuber (1997) for context modeling), and with dependency-word based embeddings Bansal et al. (2014); Melamud et al. (2016b) lead to similar results and same conclusions.

**AR Specialisation and Constraints (X\({}_{s}\rightarrow\) X\({}^{\prime}_{s}\))** We experiment with linguistic constraints used before by Mrksic et al. (2017); Vulic et al. (2017): they extracted monolingual synonymy/attract pairs from the Paraphrase Database (PPDB) Ganitkevitch et al. (2013); Pavlick et al. (2015) (640,435 synonymy pairs in total), while their antonymy/repel constraints came from BabelNet Navigli and Ponzetto (2012) (11,939 pairs).5

Footnote 5: We have experimented with another set of constraints used in prior work Zhang et al. (2014); Ono et al. (2015), reaching similar conclusions: these were extracted from WordNet Fellbaum (1998) and Roget Kipfer (2009), and comprise 1,023,082 synonymy pairs and 380,873 antonymy pairs.

The coverage of \(\mathcal{V}_{d}\) vocabulary words in the constraints illustrates well the problem of unseen words with the fine-tuning specialisation models. For instance, the constraints cover only a small subset of the entire vocabulary \(\mathcal{V}_{d}\) for gns-bow2: 16.6%. They also cover only 15.3% of the top 200K most frequent \(\mathcal{V}_{d}\) words from fastText.

**Network Design and Parameters (X\({}_{u}\rightarrow\widehat{\mathbf{X}}^{\prime}_{u}\))** The non-linear regression function \(f:\mathbb{R}^{d}\to\mathbb{R}^{d}\) is a DFFN with \(H\) hidden layers, each of dimensionality \(d_{1}=d_{2}=\ldots=d_{H}=512\) (see Fig. 1b). Non-linear activations are used in each layer and omitted only before the final output layer to enable full-range predictions (see Fig. 1b again).

The choices of non-linear activation and initialisation are guided by recent recommendations from the literature. First, we use _swish_Ramachandran et al. (2017); Elfwing et al. (2017) as nonlinearity, defined as \(swish(x)=x\cdot\text{sigmaid}(\beta x)\). We fix \(\beta=1\) as suggested by Ramachandran et al. (2017).6 Second, we use the he normal initialisation He et al. (2015), which is preferred over the xavier initialisation Glorot and Bengio (2010) for deep models Mishkin and Matas (2016); Li et al. (2016), although in our experiments we do not observe a significant difference in performance between the two alternatives. We set \(H=5\) in all experiments without any fine-tuning; we also analyse the impact of the network depth in Sect. 5.

Footnote 6: According to Ramachandran et al. (2017), for deep networks _swish_ has a slight edge over the family of LU/ReLUated activations Maas et al. (2013); He et al. (2015); Klambauer et al. (2017). We also observe a minor (and insignificant) difference in favour of _swish_.

OptimisationFor the ar specialisation step, we adopt the original suggested model setup. Hyperparameter values are set to: \(\delta_{att}=0.6\), \(\delta_{rep}=0.0\), \(\lambda_{reg}=10^{-9}\)Mrksic et al. (2017). The models are trained for 5 epochs with Adagrad Duchi et al. (2011), with batch sizes set to \(b_{att}=b_{rep}=50\), again as in the original work.

For training the non-linear mapping with DFFN (Fig. 1b), we use the Adam algorithm Kingma and Ba (2015) with default settings. The model is trained for 100 epochs with early stopping on a validation set. We reserve 10% of all available seen data (i.e., the words from \(\mathcal{V}_{s}\) represented in X\({}_{s}\) and X\({}^{\prime}_{s}\)) for validation, the rest are used for training. For the mm objective, we set \(\delta_{mm}=0.6\) and \(k=25\) in all experiments without any fine-tuning.

## 5 Results and Discussion

### Intrinsic Evaluation: Word Similarity

Evaluation ProtocolThe first set of experiments evaluates vector spaces with different specialisation procedures intrinsically on word similarity benchmarks: we use the SimLex-999 dataset Hill et al. (2015), and SimVerb-3500 Gerz et al. (2016), a recent verb pair similarity dataset providing similarity ratings for 3,500 verb pairs.7 Spearman's

rank correlation is used as the evaluation metric.

Footnote 1: The results of the _hotl-out_ experiments on SimLex-999 and SimVerb-3500 after applying our non-linear vector space transformation with different depths (hidden layer size \(H\), see Fig. 0(b)). The results are presented as averages over 20 runs with the nonlinear-mm variant, the shaded regions are spanned by the maximum and minimum scores obtained. Thick horizontal lines refer to Spearman’s rank correlations achieved in the initial space \({\bf X}_{d}\). \(H=0\) denotes the standard _linear_ regression model Mikolov et al. (2013); Lazaridou et al. (2015) (linear-mm shown since it outperforms linear-mse).

#### Results and Analysis

The results with the three word vector collections are provided in Tab. 1. In addition, Fig. 2 plots the influence of the network depth \(H\) on the model's performance.

The results suggest that the mapping of unseen words is universally useful, as the highest correlation scores are obtained with the final fully specialised vector space \({\bf X}_{f}\) for all three input spaces. The results in the _hold-out_ setup are particularly indicative of the improvement achieved by our postspecialisation method. For instance, it achieves a +0.2 correlation gain with glove on both SimLex and SimVerb by specialising vector representations for words present in these datasets _without_ seeing a single external constraint which contains any of these words. This suggests that the perturbation of the seen subspace \({\bf X}_{s}\) by attract-repel contains implicit knowledge that can be propagated to \({\bf X}_{u}\), learning better representations for unseen words. We observe small but consistent improvements across the board in the _all_ setup. The smaller gains can be explained by the fact that a majority of

\begin{table}
\begin{tabular}{p{48.4pt} p{27.6pt} p{27.6pt} p{27.6pt} p{27.6pt} p{27.6pt} p{27.6pt} p{27.6pt} p{27.6pt} p{27.6pt} p{27.6pt} p{27.6pt}} \hline \hline  & \multicolumn{6}{c}{Setup: _bold-out_} & \multicolumn{6}{c}{Setup: _all_} \\ \cline{2-13}  &glove & spans-bow2 & fastText &glove & spans-bow2 & fastText \\ \cline{2-13}  & SL & SV & SL & SV & SL & SV & SL & SV & SL & SV \\ \cline{2-13}
**Distributional: \({\bf X}_{d}\)** & 408 & .286 & .414 & .275 & .383 & .255 & .408 & .286 & .414 & .275 & .383 & .255 \\
**+AR specialisation: \({\bf X}^{\prime}_{s}\)** & 408 & .286 & .414 & .275 & .383 & .255 & .690 & .578 & .658 & .544 & .629 & .502 \\
**++Mapping unseen: \({\bf X}_{f}\)** & 504 & .384 & .447 & .309 & .405 & .285 & .690 & .578 & .656 & .551 & .628 & .502 \\ LINEAR-MSE & .549 & .407 & .484 & .344 & .459 & .329 & .694 & .586 & .663 & .556 & .631 & .506 \\ LINEAR-MM & .548 & .422 & .468 & .329 & .419 & .308 & .697 & .582 & .663 & .554 & .628 & .487 \\ Nonlinear-mm & **.603** & **.480** & **.531** & **.391** & **.471** & **.349** & **.705** & **.600** & **.667** & **.562** & **.638** & **.507** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Spearman’s \(\rho\) correlation scores for three word vector collections on two English word similarity datasets, SimLex-999 (SL) and SimVerb-3500 (SV), using different mapping variants, evaluation protocols, and word vector spaces: from the initial distributional space \({\bf X}_{d}\) to the fully specialised space \({\bf X}_{f}\). \(H=5\).

Figure 2: The results of the _hold-out_ experiments on SimLex-999 and SimVerb-3500 after applying our non-linear vector space transformation with different depths (hidden layer size \(H\), see Fig. 0(b)). The results are presented as averages over 20 runs with the nonlinear-mm variant, the shaded regions are spanned by the maximum and minimum scores obtained. Thick horizontal lines refer to Spearman’s rank correlations achieved in the initial space \({\bf X}_{d}\). \(H=0\) denotes the standard _linear_ regression model Mikolov et al. (2013); Lazaridou et al. (2015) (linear-mm shown since it outperforms linear-mse).

 

SimLex and SimVerb words are present in the external constraints (93.7% and 87.2%, respectively).

The scores also indicate that _both_ non-linearity and the chosen objective function contribute to the quality of the learned mapping: largest gains are reported with the nonlinear-mm variant which **a)** employs non-linear activations and **b)** replaces the basic mean-squared-error objective with maxmargin. The usefulness of the latter has been established in prior work on cross-space mapping learning Lazaridou et al. (2015). The former indicates that the initial ar transformation is non-linear. It is guided by a large number of constraints; their effect cannot be captured by a simple linear map as in prior work on, e.g., cross-lingual word embeddings Mikolov et al. (2013); Ruder et al. (2017).

Finally, the analysis of the network depth \(H\) indicates that going deeper helps only to a certain extent. Adding more layers allows for a richer parametrisation of the network (which is beneficial given the number of linguistic constraints used by ar). This makes the model more expressive, but it seems to saturate with larger \(H\) values.

Post-Specialisation with Other Post-ProcessorsWe also verify that our post-specialisation approach is not tied to the attract-repel method, and is indeed applicable on top of any post-processing specialisation method. We analyse the impact of post-specialisation in the _hold-out_ setting using the original _retrofitting_ (_RFit_) model Faruqui et al. (2015) and _counter-fitting_ (_CFit_) Mrksic et al. (2016) in lieu of attract-repel. The results on word similarity with the best-performing nonlinear-mm variant are summarised in Tab. 2.

The scores again indicate the usefulness of post-specialisation. As expected, the gains are lower than with attract-repel. _RFit_ falls short of _CFit_ as by design it can leverage only synonymy (i.e., attract) external constraints.

### Downstream Task I: DST

Next, we evaluate the usefulness of post-specialisation for two downstream tasks - dialogue state tracking and lexical text simplification - in which discerning semantic similarity from other types of semantic relatedness is crucial. We first evaluate the importance of post-specialisation for a downstream language understanding task of _dialogue state tracking_ (DST) Henderson et al. (2014); Williams et al. (2016), adopting the evaluation protocol and data of Mrksic et al. (2017).

Dst: Model and EvaluationThe DST model is the first component of modern dialogue pipelines Young (2010), which captures the users' goals at each dialogue turn and then updates the dialogue state. Goals are represented as sets of constraints expressed as slot-value pairs (e.g., food=_Chinese_). The set of slots and the set of values for each slot constitute the ontology of a dialogue domain. The probability distribution over the possible states is the system's estimate of the user's goals, and it is used by the dialogue manager module to select the subsequent system response Su et al. (2016). An example in Fig. 3 illustrates the DST pipeline.

For evaluation, we use the Neural Belief Tracker (NBT), a state-of-the-art DST model which was the first to reason purely over pre-trained word vectors Mrksic et al. (2017).8 The NBT uses no hand-crafted semantic lexicons, instead composing word vectors into intermediate utterance and context representations.9 For full model details, we refer the reader to the original paper. The importance of word vector specialisation for the DST task (e.g., distinguishing between synonyms and antonyms by pulling _northern_ and _north_ closer in

Figure 3: DST labels (user goals given by slot-value pairs) in a multi-turn dialogue Mrksić et al. (2015).

 

the vector space while pushing _north_ and _south_ away) has been established (Mrksic et al., 2017).

Again, as in prior work the DST evaluation is based on the Wizard-of-Oz (WOZ) v2.0 dataset (Wen et al., 2017; Mrksic et al., 2017), comprising 1,200 dialogues split into training (600 dialogues), development (200), and test data (400). In all experiments, we report the standard DST performance measure: _joint goal accuracy_, and report scores as averages over 5 NBT training runs.

Results and AnalysisWe again evaluate word vectors in two settings: **1)**_hold-out_, where linguistic constraints with words appearing in the WOZ data are removed, making all WOZ words unseen by attract-repel; and **2)**_all_. The results for the English DST task with different glove word vector variants are summarised in Tab. 3; similar trends in results are observed with two other word vector collections. The scores maintain conclusions established in the word similarity task. First, semantic specialisation with attract-repel is again beneficial, and discerning between synonyms and antonyms improves DST performance. However, specialising unseen words (the final \({\bf X}_{u}\) vector space) yields further improvements in both evaluation settings, supporting our claim that the specialisation signal can be propagated to unseen words.

This downstream evaluation again demonstrates the importance of non-linearity, as the peak scores are reported with the nonlinear-mm variant. More substantial gains in the _all_ setup are observed in the DST task compared to the word similarity task. This stems from a lower coverage of the WOZ data in the ar constraints: 36.3% of all WOZ words are unseen words. Finally, the scores are higher on average in the _all_ setup, since this setup uses more external constraints for ar, and consequently uses more training examples to learn the mapping.

Other LanguagesWe test the portability of our framework to two other languages for which we have similar evaluation data: German (DE) and Italian (IT). SimLex-999 has been translated and rescored in the two languages by Leviant and Reichart (2015), and the WOZ data were translated and adapted by Mrksic et al. (2017). Exactly the same setup is used as in our English experiments, without any additional language-specific fine-tuning. Linguistic constraints were extracted from the same sources: synonyms from the PPDB (135,868 in DE, 362,452 in IT), antonyms from BabelNet (4,124 in DE, and 16,854 in IT). Our starting distributional vector spaces are taken from prior work: IT vectors are from (Dinu et al., 2015), DE vectors are from (Vulic and Korhonen, 2016a). The results are summarised in Tab. 4.

Our post-specialisation approach yields consistent improvements over the initial distributional space and the ar specialisation model in both tasks and for both languages. We do not observe any gain or IT SimLex in the _all_ setup since IT constraints have almost complete coverage of all IT SimLex words (99.3%; the coverage is 64.8% in German). As expected, the DST scores in the _all_ setup are higher than in the _hold-out_ setup due to a larger number of constraints and training examples.

Lower absolute scores for Italian and German compared to the ones reported for English are due to multiple factors, as discussed previously by Mrksic et al. (2017): **1)** the ar model uses less linguistic constraints for DE and IT; **2)** distributional word vectors are induced from smaller corpora; **3)**inguistic phenomena (e.g., cases and compounding in DE) contribute to data sparsity and also make the DST task more challenging. However, it is important to stress the consistent gains over the vector space specialised by the state-of-the-art attract-repel model across all three test languages. This indicates that the proposed approach is language-agnostic and portable to multiple languages.

### Downstream Task II: Lexical Simplification

In our second downstream task, we examine the effects of post-specialisation on lexical simplification (LS) in English. LS aims to substitute complex words (i.e., less commonly used words) with their simpler synonyms in the context. Simplified text must keep the meaning of the original text, which is discerning similarity from relatedness is important (e.g., in "_The automobile was set on fire"_ the word _automobile"_ should be replaced with _"car"_ or _"vehicle"_ but not with _"wheel"_ or _"driver"_).

\begin{table}
\begin{tabular}{l l l} \hline \hline English & _hold-out_ & _all_ \\
**Distributional:**\({\bf X}_{d}\) & .797 & .797 \\
**+Ar Spec.:**\({\bf X}^{*}_{u}\cup{\bf X}_{u}\) & .797 & .817 \\
**++Mapping:**\({\bf X}_{f}={\bf X}^{*}_{t}\cup\widehat{{\bf X}^{*}_{t}}\) & & \\ linear-mm & .815 & .818 \\ Nonlinear-mm & **.827** & **.835** \\ \hline \end{tabular}
\end{table}
Table 3: DST results in two evaluation settings (_hold-out_ and _all_) with different glove variants.

 

We employ Light-LS (Glavas and Stajner, 2015), a lexical simplification algorithm that: 1) makes substitutions based on word similarities in a semantic vector space, and 2) can be provided an arbitrary embedding space as input.10 For a complex word, Light-LS considers the most similar words from the vector space as simplification candidates. Candidates are ranked according to several features, indicating simplicity and fitness for the context (semantic relatedness to the context of the complex word). The substitution is made if the best candidate is simpler than the original word. By providing vector spaces post-specialised for semantic similarity to Light-LS, we expect to more often replace complex words with their true synonyms.

Footnote 10: [https://github.com/codogogo/lights](https://github.com/codogogo/lights)

We evaluate Light-LS performance in the _all_ setup on the LS benchmark compiled by Horn et al. (2014), who crowdsourced 50 manual simplifications for each complex word. As in prior work, we evaluate performance with the following metrics: 1) _Accuracy_ (Acc.) is the number of correct simplifications made (i.e., the system made the simplification and its substitution is found in the list of crowdsourced substitutions), divided by the total number of indicated complex words; 2) _Changed_ (Ch.) is the percentage of indicated complex words that were replaced by the system (whether or not the replacement was correct).

LS results are summarised in Tab. 5. Post-specialised vector spaces consistently yield 5-6% gain in _Accuracy_ compared to respective distributional vectors and embeddings specialised with the state-of-the-art Attract-Repel model. Similar to DST evaluation, improvements over Attract-Repel demonstrate the importance of specialising the vectors of the entire vocabulary and not only the vectors of words from the external constraints.

## 6 Conclusion and Future Work

We have presented a novel post-processing model, termed _post-specialisation_, that specialises word vectors for the full vocabulary of the input vector space. Previous post-processing specialisation models fine-tune word vectors only for words occurring in external lexical resources. In this work, we have demonstrated that the specialisation of the subspace of seen words can be leveraged to learn a mapping function which specialises vectors for all other words, unseen in the external resources. Our results across word similarity and downstream language understanding tasks show consistent improvements over the state-of-the-art specialisation method for all three test languages.

In future work, we plan to extend our approach to specialisation for asymmetric relations such as hypernymy or meronymy (Glavas and Ponzetto, 2017; Nickel and Kiela, 2017; Vulic and Mrksic, 2018). We will also investigate more sophisticated non-linear functions. The code is available at: [https://github.com/cambridgeltl/post-specialisation/](https://github.com/cambridgeltl/post-specialisation/).

## Acknowledgments

We thank the three anonymous reviewers for their insightful suggestions. This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909).

\begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline  & \multicolumn{3}{c}{German} & \multicolumn{3}{c}{Italian} \\ \cline{2-9}  & SimLex (Similarity) & \multicolumn{1}{l}{WOZ (DST)} & \multicolumn{1}{l}{SimLex (Similarity)} & \multicolumn{1}{l}{WOZ (DST)} \\  & _hold-out_ & _all_ & _hold-out_ & _all_ & _hold-out_ & _all_ & _hold-out_ & _all_ \\ \cline{2-9}
**Distributional: X\({}_{d}\)** & .267 & .267 & .487 & .487 & .363 & .363 & .618 & .618 \\
**+Ar Spec.: X\({}^{\prime}_{a}\cup\)X\({}_{u}\)** & .267 & .422 & .487 & .535 & .363 & **.616** & .618 & .634 \\
**++Mapping: X\({}_{f}\)** & .354 & .449 & .485 & .533 & .401 & **.616** & .627 & .633 \\
**nonlinear-mm** & **.367** & **.466** & **.496** & **.538** & **.428** & **.616** & **.637** & **.647** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results on word similarity (Spearman’s \(\rho\)) and DST (joint goal accuracy) for German and Italian.

\begin{table}
\begin{tabular}{l l l l} \hline \hline Vectors & Specialisation & Acc. & Ch. \\ \hline \multirow{3}{*}{Glove} & **Distributional:** X\({}_{d}\) & 66.0 & **94.0** \\  & **+AR Spec.:** X\({}^{\prime}_{a}\cup\)X\({}_{u}\) & 67.6 & 87.0 \\  & **+Mapping: X\({}_{f}\)** & **72.3** & 87.6 \\ \hline \multirow{3}{*}{fastText} & **Distributional:** X\({}_{d}\) & 57.8 & 84.0 \\  & **+AR Spec.:** X\({}^{\prime}_{a}\cup\)X\({}_{u}\) & 69.8 & **89.4** \\  & **+MPping: X\({}_{f}\)** & **74.3** & 88.8 \\ \hline \multirow{3}{*}{sgnS-bow2} & **Distributional:** X\({}_{d}\) & 56.0 & 79.1 \\  & **+AR Spec.:** X\({}^{\prime}_{a}\cup\)X\({}_{u}\) & 64.4 & 86.7 \\ \cline{1-1}  & **+AR Mapping: X\({}_{f}\)** & **70.9** & **86.8** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Lexical simplification performance with post-specialisation applied on three input spaces.

 



## References

* Al-Rfou et al. (2013) Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In _Proceedings of CoNLL_, pages 183-192.
* Artetxe et al. (2016) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In _Proceedings of EMNLP_, pages 2289-2294.
* Artetxe et al. (2017) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In _Proceedings of ACL_, pages 451-462.
* Bansal et al. (2014) Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In _Proceedings of ACL_, pages 809-815.
* Bian et al. (2014) Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014. Knowledge-powered deep learning for word embedding. In _Proceedings of ECML-PKDD_, pages 132-148.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the ACL_, 5:135-146.
* Bruni et al. (2014) Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. _Journal of Artificial Intelligence Research_, 49:1-47.
* Chen and Manning (2014) Danqi Chen and Christopher D. Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proceedings of EMNLP_, pages 740-750.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* Conneau et al. (2017) Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou. 2017. Word translation without parallel data. _CoRR_, abs/1710.04087.
* Cotterell et al. (2016) Ryan Cotterell, Hinrich Schutze, and Jason Eisner. 2016. Morphological smoothing and extrapolation of word embeddings. In _Proceedings of ACL_, pages 1651-1660.
* Dinu et al. (2015) Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2015. Improving zero-shot learning by mitigating the hubness problem. In _Proceedings of ICLR (Workshop Papers)_.
* Duchi et al. (2011) John C. Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _Journal of Machine Learning Research_, 12:2121-2159.
* Elfwing et al. (2017) Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2017. Sigmod-weighted linear units for neural network function approximation in reinforcement learning. _CoRR_, abs/1702.03118.
* Faruqui (2016) Manaal Faruqui. 2016. _Diverse Context for Learning Word Representations_. Ph.D. thesis, Carnegie Mellon University.
* Faruqui et al. (2015) Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In _Proceedings of NAACL-HLT_, pages 1606-1615.
* Fellbaum (1998) Christiane Fellbaum. 1998. _WordNet_.
* Finkelstein et al. (2002) Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadil Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. _ACM Transactions on Information Systems_, 20(1):116-131.
* Frome et al. (2013) Andrea Frome, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. 2013. DeViSE: A deep visual-semantic embedding model. In _Proceedings of NIPS_, pages 2121-2129.
* Ganitkevitch et al. (2013) Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The Paraphrase Database. In _Proceedings of NAACL-HLT_, pages 758-764.
* Gerz et al. (2016) Daniela Gerz, Ivan Vulic, Felix Hill, Roi Reichart, and Anna Kornonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. In _Proceedings of EMNLP_, pages 2173-2182.
* Glavas and Ponzetto (2017) Goran Glavas and Simone Paolo Ponzetto. 2017. Dual tensor model for detecting asymmetric lexicosemantic relations. In _Proceedings of EMNLP_, pages 1758-1768.
* Glavas and Stajnen (2015) Goran Glavas and Sanja Stajnen. 2015. Simplifying lexical simplification: Do we need simplified corpora? In _Proceedings of ACL_, pages 63-68.
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _Proceedings of AISTATS_, pages 249-256.
* Harris (1954) Zellig S. Harris. 1954. Distributional structure. _Word_, 10(23):146-162.
* He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In _Proceedings of ICCV_, pages 1026-1034.
* Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Jason D. Wilians. 2014. The Second Dialog State Tracking Challenge. In _Proceedings of SIGDIAL_, pages 263-272

Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. _Computational Linguities_, 41(4):665-695.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural Computation_, 9(8):1735-1780.
* Horn et al. (2014) Colby Horn, Cathryn Manduca, and David Kauchak. 2014. Learning a lexical simplifier using wikipedia. In _Proceedings of the ACL_, pages 458-463.
* Kiela et al. (2015) Douwe Kiela, Felix Hill, and Stephen Clark. 2015. Specializing word embeddings for similarity or relatedness. In _Proceedings of EMNLP_, pages 2044-2048.
* Kim et al. (2016) Joo-Kyung Kim, Gokhan Tur, Asli Celikyilmaz, Bin Cao, and Ye-Yi Wang. 2016. Intent detection using semantically enriched word embeddings. In _Proceedings of SLT_.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of ICLR (Conference Track)_.
* Kipfer (2009) Barbara Ann Kipfer. 2009. _Roget's 21st Century Thesaurus (3rd Edition)_. Philip Lief Group.
* Klambauer et al. (2017) Gunter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. _CoRR_, abs/1706.02515.
* Kummerfeld et al. (2015) Jonathan K. Kummerfeld, Taylor Berg-Kirkpatrick, and Dan Klein. 2015. An empirical analysis of optimization for max-margin NLP. In _Proceedings of EMNLP_, pages 273-279.
* Lazaridou et al. (2015) Angeliki Lazaridou, Georgiana Dinu, and Marco Baroni. 2015. Hubness and pollution: Delving into cross-space mapping for zero-shot learning. In _Proceedings of ACL_, pages 270-280.
* Leviant and Reichart (2015) Ira Leviant and Roi Reichart. 2015. Separated by an un-common language: Towards judgment language informed vector space modeling. _CoRR_, abs/1508.00106.
* Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In _Proceedings of ACL_, pages 302-308.
* Levy et al. (2015) Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. _Transactions of the ACL_, 3:211-225.
* Li et al. (2016) Sihan Li, Jiantoao Jiao, Yanjun Han, and Tsachy Weissman. 2016. Demystifying ResNet. _CoRR_, abs/1611.01186.
* Liu et al. (2015) Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, and Yu Hu. 2015. Learning semantic word embeddings based on ordinal knowledge constraints. In _Proceedings of ACL_, pages 1501-1511.
* Maas et al. (2013) Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In _Proceedings of ICML_.
* Melamud et al. (2016a) Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016a. Context2vec: Learning generic context embedding with bidirectional LSTM. In _Proceedings of CoNLL_, pages 51-61.
* Melamud et al. (2016b) Oren Melamud, David McClosky, Siddharith Patwardhan, and Mohit Bansal. 2016b. The role of context types and dimensionality in learning word embeddings. In _Proceedings of NAACL-HLT_, pages 1030-1040.
* Mikolov et al. (2013a) Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013a. Exploiting similarities among languages for machine translation. _arXiv preprint, CoRR_, abs/1309.4168.
* Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In _Proceedings of NIPS_, pages 3111-3119.
* Mishkin and Matas (2016) Dmytro Mishkin and Jiri Matas. 2016. All you need is a good init. In _Proceedings of ICLR (Conference Track)_.
* Mrksic et al. (2017) Nikola Mrksic, Diarmuid O Seaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In _Proceedings of ACL_, pages 1777-1788.
* Mrksic et al. (2015) Nikola Mrksic, Diarmuid O Seaghdha, Blaise Thomson, Milica Gasic, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2015. Multidomain dialog state tracking using recurrent neural networks. In _Proceedings of ACL_, pages 794-799.
* Mrksic et al. (2016) Nikola Mrksic, Diarmuid O Seaghdha, Blaise Thomson, Milica Gasic, Lina Maria Rojas-Barahona, Peif-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In _Proceedings of NAACL-HLT_.
* Mrksic et al. (2017) Nikola Mrksic, Ivan Vulic, Diarmuid O Seaghdha, Ira Leviant, Roi Reichart, Milica Gasic, Anna Kornonen, and Steve Young. 2017. Semantic specialisation of distributional word vector spaces using monolingual and cross-lingual constraints. _Transactions of the ACL_, 5:309-324.
* Nair and Hinton (2010) Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In _Proceedings of ICML_, pages 807-814.
* Navigli and Ponzetto (2012) Roberto Navigli and Simone Paolo Ponzetto. 2012. BaghelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. _Artificial Intelligence_, 193:217-250 

Kim Anh Nguyen, Maximilian Koper, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017. Hierarchical embeddings for hypernymy detection and directionality. In _Proceedings of EMNLP_, pages 233-243.
* Nguyen et al. (2016) Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2016. Integrating distributional lexical contrast into word embeddings for antonymynM distinction. In _Proceedings of ACL_, pages 454-459.
* Nickel and Kiela (2017) Maximilian Nickel and Douwe Kiela. 2017. Poincare embeddings for learning hierarchical representations. In _Proceedings of NIPS_.
* Ono et al. (2015) Masataka Ono, Makoto Miwa, and Yutaka Sasaki. 2015. Word embedding-based antonym detection using thesauri and distributional information. In _Proceedings of NAACL-HLT_, pages 984-989.
* Osborne et al. (2016) Dominique Osborne, Shashi Narayan, and Shay Cohen. 2016. Encoding prior knowledge with eigenword embeddings. _Transactions of the ACL_, 4:417-430.
* Pavlick et al. (2015) Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In _Proceedings of ACL_, pages 425-430.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of EMNLP_, pages 1532-1543.
* Ramachandran et al. (2017) Prajit Ramachandran, Barret Zoph, and Quoc V. Le. 2017. Searching for activation functions. _CoRR_, abs/1710.05941.
* Rothe and Schutze (2015) Sascha Rothe and Hinrich Schutze. 2015. AutoExtend: Extending word embeddings to embeddings for synsets and lexemes. In _Proceedings of ACL_, pages 1793-1803.
* Ruder et al. (2017) Sebastian Ruder, Ivan Vulic, and Anders Suggaard. 2017. A survey of cross-lingual embedding models. _CoRR_, abs/1706.04902.
* Schwartz et al. (2015) Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015. Symmetric pattern based word embeddings for improved word similarity prediction. In _Proceedings of CoNLL_, pages 258-267.
* Su et al. (2016) Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Continuously learning neural dialogue management. In _arXiv preprint: 1606.02689_.
* Vulic and Karbonen (2016a) Ivan Vulic and Anna Karbonen. 2016a. Is "universal syntax" universally useful for learning distributed word representations? In _Proceedings of ACL_, pages 518-524.
* Vulic and Korhonen (2016b) Ivan Vulic and Anna Korhonen. 2016b. On the role of seed lexicons in learning bilingual word embeddings. In _Proceedings of ACL_, pages 247-257.
* Vulic and Mrksic (2018) Ivan Vulic and Nikola Mrksic. 2018. Specialising word vectors for lexical entailment. In _Proceedings of NAACL-HLT_.
* Vulic et al. (2017a) Ivan Vulic, Nikola Mrksic, and Anna Korhonen. 2017a. Cross-lingual induction and transfer of verb classes based on word vector space specialisation. In _Proceedings of EMNLP_, pages 2536-2548.
* Vulic et al. (2017b) Ivan Vulic, Nikola Mrksic, Roi Reichart, Diarmuid O Seaghdha, Steve Young, and Anna Korhonen. 2017b. Morph-fitting: Fine-tuning word vector spaces with simple language-specific rules. In _Proceedings of ACL_, pages 56-68.
* Wen et al. (2017) Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In _Proceedings of EACL_.
* Weston et al. (2011) Jason Weston, Samy Bengio, and Nicolas Usunier. 2011. WSABIE: Scaling up to large vocabulary image annotation. In _Proceedings of IJCAI_, pages 2764-2770.
* Wieting et al. (2015) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. _Transactions of the ACL_, 3:345-358.
* Williams et al. (2016) Jason D. Williams, Antoine Raux, and Matthew Henderson. 2016. The Dialog State Tracking Challenge series: A review. _Dialogue & Discourse_, 7(3):4-33.
* Xu et al. (2014) Chang Xu, Yalong Bai, Jiang Bian, Bin Gao, Gang Wang, Xiaoguang Liu, and Tie-Yan Liu. 2014. RCNET: A general framework for incorporating knowledge into word representations. In _Proceedings of CIKM_, pages 1219-1228.
* Young (2010) Steve Young. 2010. Cognitive User Interfaces. _IEEE Signal Processing Magazine_.
* Yu and Dredze (2014) Mo Yu and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In _Proceedings of ACL_, pages 545-550.
* Zhang et al. (2014) Jingwei Zhang, Jeremy Salwen, Michael Glass, and Alfio Glozzo. 2014. Word semantic representations using bayesian probabilistic tensor factorization. In _Proceedings of EMNLP_, pages 1522-1531.