Venue: NAACL 2016

# Counter-fitting Word Vectors to Linguistic Constraints

 Nikola Mrksic\({}^{1}\),   Diarmuid O Seaghdha\({}^{2}\),    Blaise Thomson\({}^{2}\),  Milica Gasic\({}^{1}\)

**Lina Rojas-Barahona\({}^{1}\), Pei-Hao Su\({}^{1}\),   David Vandyke\({}^{1}\),   Tsung-Hsien Wen\({}^{1}\),   Steve Young\({}^{1}\)**

\({}^{1}\) Department of Engineering, University of Cambridge, UK

\({}^{2}\) Apple Inc.

{nm480,mg436,phs26,dy27,thw28,sjy}@cam.ac.uk

{doseaghdha, blaisethom}@apple.com

###### Abstract

In this work, we present a novel _counter-fitting_ method which injects antonymy and synonymy constraints into vector space representations in order to improve the vectors' capability for judging semantic similarity. Applying this method to publicly available pre-trained word vectors leads to a new state of the art performance on the SimLex-999 dataset. We also show how the method can be used to tailor the word vector space for the downstream task of dialogue state tracking, resulting in robust improvements across different dialogue domains.

## 1 Introduction

Many popular methods that induce representations for words rely on the _distributional hypothesis_ - the assumption that semantically similar or related words appear in similar contexts. This hypothesis supports unsupervised learning of meaningful word representations from large corpora Curran (2003); O Seaghdha and Korhonen (2014); Mikolov et al. (2013); Pennington et al. (2014). Word vectors trained using these methods have proven useful for many downstream tasks including machine translation Zou et al. (2013) and dependency parsing Bansal et al. (2014).

One drawback of learning word embeddings from co-occurrence information in corpora is that it tends to coalesce the notions of _semantic similarity_ and _conceptual association_Hill et al. (2014b). Furthermore, even methods that can distinguish similarity from association (e.g., based on syntactic co-occurrences) will generally fail to tell synonyms from antonyms Mohammad et al. (2008). For example, words such as _east_ and _west_ or _expensive_ and _inexpensive_ appear in near-identical contexts, which means that distributional models produce very similar word vectors for such words. Examples of such anomalies in GloVe vectors can be seen in Table 1, where words such as _cheaper_ and _inexpensive_ are deemed similar to (their antonym) _expensive_.

A second drawback is that similarity and antonymy can be application- or domain-specific. In our case, we are interested in exploiting distributional knowledge for the _dialogue state tracking_ task (DST). The DST component of a dialogue system is responsible for interpreting users' utterances and updating the system's _belief state_ - a probability distribution over all possible states of the dialogue. For example, a DST for the restaurant domain needs to detect whether the user wants a _cheap_ or _expensive_ restaurant. Being able to generalise using distributional information while still distinguishing between semantically different yet conceptually related words

\begin{table}
\begin{tabular}{c|c c c}  & **east** & **expensive** & **British** \\ \hline \multirow{5}{*}{Before} & west & pricey & American \\  & north & cheaper & Australian \\  & south & costly & Britain \\  & southeast & overpriced & European \\  & northeast & inexpensive & England \\ \hline \multirow{5}{*}{After} & eastward & costly & Brits \\  & eastern & pricy & London \\ \cline{1-1}  & easterly & overpriced & BBC \\ \cline{1-1}  & - & pricey & UK \\ \cline{1-1}  & - & afford & Britain \\ \end{tabular}
\end{table}
Table 1: Nearest neighbours for target words using GloVe vectors before and after counter-fitting

(e.g. _cheaper_ and _pricey_) is critical for the performance of dialogue systems. In particular, a dialogue system can be led seriously astray by false synonyms.

We propose a method that addresses these two drawbacks by using synonymy and antonymy relations drawn from either a general lexical resource or an application-specific ontology to fine-tune distributional word vectors. Our method, which we term _counter-fitting_, is a lightweight post-processing procedure in the spirit of _retrofitting_Faruqui et al. (2015). The second row of Table 1 illustrates the results of counter-fitting: the nearest neighbours capture true similarity much more intuitively than the original GloVe vectors. The procedure improves word vector quality regardless of the initial word vectors provided as input.1 By applying counter-fitting to the Paragram-SL99W word vectors provided by Wieting et al. (2015), we achieve new state-of-the-art performance on SimLex-999, a dataset designed to measure how well different models judge semantic similarity between words Hill et al. (2014b). We also show that the counter-fitting method can inject knowledge of dialogue domain ontologies into word vector space representations to facilitate the construction of semantic dictionaries which improve DST performance across two different dialogue domains. Our tool and word vectors are available at github.com/nmrksic/counter-fitting.

Footnote 1: When we write “improve”, we refer to improving the vector space for a specific purpose. We do not expect that a vector space fine-tuned for semantic similarity will give better results on semantic relatedness. As Mohammad et al. (2008) observe, antonymous concepts are related but not similar.

## 2 Related Work

Most work on improving word vector representations using lexical resources has focused on bringing words which are known to be semantically related closer together in the vector space. Some methods modify the prior or the regularization of the original training procedure Yu and Dredze (2014); Bian et al. (2014); Kiela et al. (2015). Wieting et al. (2015) use the Paraphrase Database Ganitkevitch et al. (2013) to train word vectors which emphasise word similarity over word relatedness. These word vectors achieve the current state-of-the-art performance on the SimLex-999 dataset and are used as input for counter-fitting in our experiments.

Recently, there has been interest in lightweight post-processing procedures that use lexical knowledge to refine off-the-shelf word vectors without requiring large corpora for (re-)training as the aforementioned "heavyweight" procedures do. Faruqui et al.'s (2015) _retrofitting_ approach uses similarity constraints from WordNet and other resources to pull similar words closer together.

The complications caused by antonymy for distributional methods are well-known in the semantics community. Most prior work focuses on extracting antonym pairs from text rather than exploiting them Lin et al. (2003); Mohammad et al. (2008); Turney (2008); Hashimoto et al. (2012); Mohammad et al. (2013). The most common use of antonym information is to provide features for systems that detect contradictions or logical entailment Marcu and Echihabi (2002); de Marneffe et al. (2008); Zanzotto et al. (2009). As far as we are aware, there is no previous work on exploiting antonymy in dialogue systems. The modelling work closest to ours are Liu et al. (2015), who use antonymy and WordNet hierarchy information to modify the heavyweight Word2Vec training objective; Yih et al. (2012), who use a Siamese neural network to improve the quality of Latent Semantic Analysis vectors; Schwartz et al. (2015), who build a standard distributional model from co-occurrences based on _symmetric patterns_, with specified antonymy patterns counted as negative co-occurrences; and Ono et al. (2015), who usehesauri and distributional data to train word embeddings specialised for capturing antonymy.

## 3 Counter-fitting Word Vectors to Linguistic Constraints

Our starting point is an indexed set of word vectors \(V=\{\mathbf{v}_{1},\mathbf{v}_{2},\ldots,\mathbf{v}_{N}\}\) with one vector for each word in the vocabulary. We will inject semantic relations into this vector space to produce new word vectors \(V^{\prime}=\{\mathbf{v}^{\prime}{}_{1},\mathbf{v}^{\prime}{}_{2},\ldots, \mathbf{v}^{\prime}{}_{N}\}\). For antonymy and synonymy we have a set of constraints \(A\) and \(S\), respectively. The elements of each set are pairs of word indices; for example, each pair \((i,j)\) in \(S\) is such that the \(i\)-th and \(j\)-th words in the vocabulary are synonyms. The objective function used to counter-fit the pre-trained word vectors \(V\) to the sets of linguistic constraints \(A\) and \(S\) contains three different terms: 



**1. Antonym Repel (AR)**: This term serves to _push_ antonymous words' vectors away from each other in the transformed vector space \(V^{\prime}\):

\[\text{AR}(V^{\prime})=\sum_{(u,w)\in A}\tau\left(\delta-d(\mathbf{v}^{\prime}_{ u},\mathbf{v}^{\prime}_{w})\right)\]

where \(d(v_{i},v_{j})=1-\cos(v_{i},v_{j})\) is a distance derived from cosine similarity and \(\tau(x)\triangleq max(0,x)\) imposes a margin on the cost. Intuitively, \(\delta\) is the "ideal" minimum distance between anonymous words; in our experiments we set \(\delta=1.0\) as it corresponds to vector orthogonality.

**2. Synonym Attract (SA)**: The counter-fitting procedure should seek to bring the word vectors of known synonymous word pairs closer together:

\[\text{SA}(V^{\prime})=\sum_{(u,w)\in S}\tau\left(d(\mathbf{v}^{\prime}_{u}, \mathbf{v}^{\prime}_{w})-\gamma\right)\]

where \(\gamma\) is the "ideal" maximum distance between synonymous words; we use \(\gamma=0\).

**3. Vector Space Preservation (VSP)**: the topology of the original vector space describes relationships between words in the vocabulary captured using distributional information from very large textual corpora. The VSP term bends the transformed vector space towards the original one as much as possible in order to preserve the semantic information contained in the original vectors:

For computational efficiency, we do not calculate distances for every pair of words in the vocabulary. Instead, we focus on the (pre-computed) neighbourhood \(N(i)\), which denotes the set of words within a certain radius \(\rho\) around the \(i\)-th word's vector in the original vector space \(V\). Our experiments indicate that counter-fitting is relatively insensitive to the choice of \(\rho\), with values between 0.2 and 0.4 showing little difference in quality; here we use \(\rho=0.2\).

The objective function for the training procedure is given by a weighted sum of the three terms:

\[C(V,V^{\prime})=k_{1}\text{AR}(V^{\prime})+k_{2}\text{SA}(V^{\prime})+k_{3} \text{VSP}(V,V^{\prime})\]

where \(k_{1},k_{2},k_{3}\geq 0\) are hyperparameters that control the relative importance of each term. In our experiments we set them to be equal: \(k_{1}=k_{2}=k_{3}\). To minimise the cost function for a set of starting vectors \(V\) and produce counter-fitted vectors \(V^{\prime}\), we run stochastic gradient descent (SGD) for 20 epochs. An end-to-end run of counter-fitting takes less than two minutes on a laptop with four CPUs.

### 3.1 Injecting Dialogue Domain Ontologies into Vector Space Representations

Dialogue state tracking (DST) models capture users' goals given their utterances. Goals are represented as sets of constraints expressed by _slot-value_ pairs such as [food: _Indian_] or [parking: _allowed_]. The set of slots \(S\) and the set of values \(V_{s}\) for each slot make up the _ontology_ of a dialogue domain.

In this paper we adopt the recurrent neural network (RNN) framework for tracking suggested in Henderson et al. (2014); Henderson et al. (2014); Mrksic et al. (2015). Rather than using a spoken language understanding (SLU) decoder to convert user utterances into meaning representations, this model operates directly on the \(n\)-gram features extracted from the automated speech recognition (ASR) hypotheses. A drawback of this approach is that the RNN model can only perform exact string matching to detect the slot names and values mentioned by the user. It cannot recognise synonymous words such as _pricey_ and _expensive_, or even subtle morphological variations such as _moderate_ and _moderately_. A simple way to mitigate this problem is to use _semantic dictionaries_: lists of rephrasings for the values in the ontology. Manual construction of dictionaries is highly labour-intensive; however, if one could automatically detect high-quality rephrasings, then this capability would come at no extra cost to the system designer.

To obtain a set of word vectors which can be used for creating a semantic dictionary, we need to _inject_ the domain ontology into the vector space. This can be achieved by introducing antonymy constraints between all the possible values of each slot (i.e. _Chinese_ and _Indian_, _expensive_ and _cheap_, etc.). The remaining linguistic constraints can come from semantic lexicons: the richer the sets of injected synonyms and antonyms are, the better the resulting word representations will become



## 4 Experiments

### Word Vectors and Semantic Lexicons

Two different collections of pre-trained word vectors were used as input to the counter-fitting procedure:

1. **Glove Common Crawl** 300-dimensional vectors made available by Pennington et al. (2014).
2. **Paragram-SL999** 300-dimensional vectors made available by Wieting et al. (2015).

The synonymy and antonym constraints were obtained from two semantic lexicons:

1. **PPDB 2.0 Pavlick et al. (2015):** the latest release of the Paraphrase Database. A new feature of this version is that it assigns relation types to its word pairs. We identify the _Equivalence_ relation with synonymy and _Exclusion_ with antonymy. We used the largest available (XXXL) version of the database and only considered single-token terms.
2. **WordNet Miller (1995)**: a well known semantic lexicon which contains vast amounts of high quality human-annotated synonym and antonym pairs. Any two words in our vocabulary which had anonymous word senses were considered antonyms; WordNet synonyms were not used.

In total, the lexicons yielded 12,802 antonymy and 31,828 synonymy pairs for our vocabulary, which consisted of 76,427 most frequent words in OpenSubtitles, obtained from invokeit.wordpress.com/frequency-word-lists/.

### Improving Lexical Similarity Predictions

In this section, we show that counter-fitting pretrained word vectors with linguistic constraints improves their usefulness for judging semantic similarity. We use Spearman's rank correlation coefficient with the SimLex-999 dataset, which contains word pairs ranked by a large number of annotators instructed to consider only semantic similarity.

Table 2 contains a summary of recently reported competitive scores for SimLex-999, as well as the performance of the unaltered, retrofitted and counter-fitted GloVe and Paragram-SL999 word vectors. To the best of our knowledge, the 0.685 figure reported for the latter represents the current high score. This figure is above the average inter-annotator agreement of 0.67, which has been referred to as the ceiling performance in most work up to now.

In our opinion, the average inter-annotator agreement is not the only meaningful measure of ceiling performance. We believe it also makes sense to compare: **a)** the model ranking's correlation with the gold standard ranking to: **b)** the average rank correlation that individual human annotators' rankings achieved with the gold standard ranking. The SimLex-999 authors have informed us that the average annotator agreement with the gold standard is 0.78.2 As shown in Table 2, the reported performance of all the models and word vectors falls well below this figure.

Footnote 2: This figure is now reported as a potentially fairer ceiling performance on the SimLex-999 website: [http://www.cl.cam.ac.uk/~fh295/simlex.html](http://www.cl.cam.ac.uk/~fh295/simlex.html).

Retrofitting pre-trained word vectors improves GloVe vectors, but not the already semantically specialised Paragram-SL999 vectors. Counter-fitting substantially improves both sets of vectors, showing that injecting antonymy relations goes a long way

\begin{table}
\begin{tabular}{|p{113.8pt}|c|} \hline
**Model / Word Vectors** & \(\rho\) \\ \hline Neural MT Model Hill et al. (2014) & 0.52 \\ Symmetric Patterns Schwartz et al. (2015) & 0.56 \\ Non-distributional Vectors Faruqui and Dyer (2015) & 0.58 \\ GloVe vectors Pennington et al. (2014) & 0.41 \\ GloVe vectors + Retrofitting GloVe + Counter-fitting Paradigm-SL999 Wieting et al. (2015) & 0.58 \\ Paragram-SL999 Stretfiting & 0.69 \\ Paragram-SL999 + Retrofitting Paragram-SL999 + Counter-fitting & **0.74** \\ \hline Inter-annotator agreement Annotatorgold standard agreement & 0.67 \\ \hline \end{tabular}
\end{table}
Table 2: Performance on SimLex-999. Retrofitting uses the code and (PPDB) data provided by the authors

\begin{table}
\begin{tabular}{|l|c|c|} \hline
**Semantic Resource** & Glove & Paramgram \\ \hline Baseline (no linguistic constraints) & 0.41 & 0.69 \\ \hline PPDB\(-\) (PPDB antonyms) & 0.43 & 0.69 \\ PPDB\(+\) (PPDB synonyms) & 0.46 & 0.68 \\ WordNet\(-\) (WordNet antonyms) & 0.52 & 0.74 \\ \hline PPDB\(-\) and PDPB\(+\) & 0.50 & 0.69 \\ WordNet\(-\) and PPBB\(-\) & 0.53 & 0.74 \\ WordNet\(-\) and PPDB\(+\) & 0.58 & 0.74 \\ \hline WordNet\(-\) and PTBD\(-\) and SPDB\(+\) & 0.58 & 0.74 \\ \hline \end{tabular}
\end{table}
Table 3: SimLex-999 performance when different sets of linguistic constraints are used for counter-fitting

towards improving word vectors for the purpose of making semantic similarity judgements.

Table 3 shows the effect of injecting different categories of linguistic constraints. GloVe vectors benefit from all three sets of constraints, whereas the quality of Paragram vectors, already exposed to PPDB, only improves with the injection of WordNet antonyms. Table 4 illustrates how incorrect similarity predictions based on the original (Paragram) vectors can be fixed through counter-fitting. The table presents eight false synonyms and nine false antonyms: word pairs with predicted rank in the top (bottom) 200 word pairs and gold standard rank 500 or more positions lower (higher). Eight of these errors are fixed by counter-fitting: the difference between predicted and gold-standard ranks is now 100 or less. Interestingly, five of the eight corrected word pairs do not appear in the sets of linguistic constraints; these are indicated by double ticks in the table. This shows that secondary (i.e. _indirect_) interactions through the three terms of the cost function do contribute to the semantic content of the transformed vector space.

### Improving Dialogue State Tracking

Table 5 shows the dialogue state tracking datasets used for evaluation. These datasets come from the Dialogue State Tracking Challenges 2 and 3 Henderson et al. (2014); Henderson et al. (2014).

We used four different sets of word vectors to construct semantic dictionaries: the original GloVe and Paragram-SL999 vectors, as well as versions counter-fitted to each domain ontology. The constraints used for counter-fitting were all those from the previous section as well as antonymy constraints among the set of values for each slot. We treated all vocabulary words within some radius \(t\) of a slot value as rephrasings of that value. The optimal value of \(t\) was determined using a grid search: we generated a dictionary and trained a model for each potential \(t\), then evaluated on the development set. Table 6 shows the performance of RNN models which used the constructed dictionaries. The dictionaries induced from the pre-trained vectors substantially improved tracking performance over the baselines (which used no semantic dictionaries). The dictionaries created using the counter-fitted vectors improved performance even further. Contrary to the SimLex-999 experiments, starting from the Paragram vectors did not lead to superior performance, which shows that injecting the application-specific ontology is at least as important as the quality of the initial word vectors.

## 5 Conclusion

We have presented a novel _counter-fitting_ method for injecting linguistic constraints into word vector space representations. The method efficiently postprocesses word vectors to improve their usefulness for tasks which involve making semantic similarity judgements. Its focus on separating vector representations of anonymous word pairs lead to substantial improvements on genuine similarity estimation tasks. We have also shown that counter-fitting can tailor word vectors for downstream tasks by using it to inject domain ontologies into word vectors used to construct semantic dictionaries for dialogue systems.

## Acknowledgements

We would like to thank Felix Hill for help with the SimLex-999 evaluation. We also thank the anonymous reviewers for their helpful suggestions.

\begin{table}
\begin{tabular}{|l|c|c|} \hline
**Word Vector Space** & **Restaurants** & **Tourist Info** \\ \hline Baseline (no dictionary) & 68.6 & 60.5 \\ \hline GloVe & 72.5 & 60.9 \\ GloVe + Counter-fitting & 73.4 & **62.8** \\ Paragram-SL999 & 73.2 & 61.5 \\ Paragram-SL999 + Counter-fitting & **73.5** & 61.9 \\ \hline \end{tabular}
\end{table}
Table 6: Performance of RNN belief trackers (ensembles of four models) with different semantic dictionaries

\begin{table}
\begin{tabular}{|p{113.8pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|} \hline

- & engage, marry & \\ \hline \end{tabular}
\end{table}
Table 4: Highest-error SimLex-999 word pairs using Paramant vectors (before counter-fitting)
 



## References

* [Bansal et al.2014] Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In _Proceedings of ACL_.
* Bian et al.2014] Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014. Knowledge-powered deep learning for word embedding. In _Machine Learning and Knowledge Discovery in Databases_.
* Curran2003] James Curran. 2003. _From Distributional to Semantic Similarity_. Ph.D. thesis, School of Informatics, University of Edinburgh.
* de Marneffe et al.2008] Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In _Proceedings of ACL_.
* Faruqui and Dyer2015] Manaal Faruqui and Chris Dyer. 2015. Non-distributional word vector representations. In _Proceedings of ACL_.
* Faruqui et al.2015] Manaal Faruqui, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting Word Vectors to Semantic Lexicons. In _Proceedings of NAACL HLT_.
* Ganitkevitch et al.2013] Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-burch. 2013. PPDB: The Paraphrase Database. In _Proceedings of NAACL HLT_.
* Hashimoto et al.2012] Chikara Hashimoto, Kentaro Torisawa, Stijn De Saeger, Jong-Hoon Oh, and Junichi Kazama. 2012. Excitatory or inhibitory: A new semantic orientation extracts contradiction and causality from the Web. In _Proceedings of EMNLP-CoNLL_.
* Henderson et al.2014a] Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014a. The Second Dialog State Tracking Challenge. In _Proceedings of SIGDIAL_.
* Henderson et al.2014b] Matthew Henderson, Blaise Thomson, and Jason D. Wilianys. 2014b. The Third Dialog State Tracking Challenge. In _Proceedings of IEEE SLT_.
* Henderson et al.2014c] Matthew Henderson, Blaise Thomson, and Steve Young. 2014c. Robust Dialog State Tracking using Delexicalised Recurrent Neural Networks and Unsupervised Adaptation. In _Proceedings of IEEE SLT_.
* Henderson et al.2014d] Matthew Henderson, Blaise Thomson, and Steve Young. 2014d. Word-Based Dialog State Tracking with Recurrent Neural Networks. In _Proceedings of SIGDIAL_.
* Hill et al.2014a] Felix Hill, Kyunghyun Cho, Sebastien Jean, Coline Devin, and Yoshua Bengio. 2014a. Embedding word similarity with neural machine translation. _Computing Research Repository_.
* Hill et al.2014b] Felix Hill, Roi Reichart, and Anna Korhonen. 2014b. SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation. _Computing Research Repository_.
* Kiela et al.2015] Douwe Kiela, Felix Hill, and Stephen Clark. 2015. Specializing word embeddings for similarity or relatedness. In _Proceedings of EMNLP_.
* Lin et al.2003] Dekang Lin, Shaojun Zhao, Lijuan Qin, and Ming Zhou. 2003. Identifying synonyms among distributionally similar words. In _Proceedings of IJCAI_.
* Liu et al.2015] Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, and Yu Hu. 2015. Learning semantic word embeddings based on ordinal knowledge constraints. In _Proceedings of ACL_.
* Marcu and Echihabi2002] Daniel Marcu and Abdsemmad Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In _Proceedings of ACL_.
* Mikolov et al.2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In _Proceedings of NIPS_.
* Miller1995] George A. Miller. 1995. WordNet: A Lexical Database for English. _Communications of the ACM_.
* Mohammad et al.2008] Saif Mohammad, Bonnie Dorr, and Graeme Hirst. 2008. Computing word-pair antonymy. In _Proceedings of EMNLP_.
* Mohammad et al.2013] Saif M. Mohammad, Bonnie J. Dorr, Graeme Hirst, and Peter D. Turney. 2013. Computing lexical contrast. _Computational Linguistics_, 39(3):555-590.
* Mrksic et al.2015] Nikola Mrksic, Diarmuid O Seaghdha, Blaise Thomson, Milica Gasic, Pei-Hao Su, David Vandyke, Tsungs-Hsien Wen, and Steve Young. 2015. Multi-domain Dialog State Tracking using Recurrent Neural Networks. In _Proceedings of ACL_.
* Ono et al.2015] Masataka Ono, Makoto Miwa, and Yutaka Sasaki. 2015. Word Embedding-based Antonym Detection using Thesauri and Distributional Information. In _Proceedings of NAACL HLT_.
* Seaghdha and Karhonen2014] Diramuid O Seaghdha and Anna Korhonen. 2014. Probabilistic distributional semantics. _Computational Linguistics_, 40(3):587-631.
* Pavlick et al.2015] Ellie Pavlick, Pushpendre Rastogi, Jiur Ganitkevich, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In _Proceedings of ACL_.
* Pennington et al.2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In _Proceedings of EMNLP_.
* Schwartz et al.2015] Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015. Symmetric pattern based word embeddings for improved word similarity prediction. In _Proceedings of CoNLL_.
* Turney2008] Peter D. Turney. 2008. A uniform approach to analogies, synonyms, antonyms, and associations. In _Proceedings of COLING_.
* Wieting et al.2015] John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. _Transactions of the Association for Computational Linguistics_

* [Yih et al.2012] Wen-Tau Yih, Geoffrey Zweig, and John C. Platt. 2012. Polarity inducing Latent Semantic Analysis. In _Proceedings of ACL_.
* [Yu and Dredze2014] Mo Yu and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In _Proceedings of ACL_.
* [Zanzotto et al.2009] Fabio Massimo Zanzotto, Marco Pennachiotti, and Alessandro Moschitti. 2009. A machine learning approach to textual entailment recognition. _Journal of Natural Language Engineering_, 15(4):551-582.
* [Zou et al.2013] Will Y. Zou, Richard Socher, Daniel M. Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In _Proceedings of EMNLP_.