Venue: NAACL 2016

# Symmetric Patterns _and_ Coordinations:

Fast _and_ Enhanced Representations of Verbs _and_ Adjectives

 Roy Schwartz,\({}^{1}\)   Roi Reichart,\({}^{2}\)   Ari Rappoport\({}^{1}\)

\({}^{1}\)Institute of Computer Science, The Hebrew University

\({}^{2}\)Faculty of Industrial Engineering and Management, Technion, IIT

{roys02|arir}@cs.huji.ac.il roiri@ie.technion.ac.il

###### Abstract

State-of-the-art word embeddings, which are often trained on bag-of-words (_BOW_) contexts, provide a high quality representation of aspects of the semantics of nouns. However, their quality decreases substantially for the task of verb similarity prediction. In this paper we show that using symmetric pattern contexts (_SPs_, e.g., "X and Y") improves word2vec verb similarity performance by up to 15% and is also instrumental in adjective similarity prediction. The unsupervised _SP_ contexts are even superior to a variety of dependency contexts extracted using a supervised dependency parser. Moreover, we observe that _SPs_ and dependency coordination contexts (_Coor_) capture a similar type of information, and demonstrate that _Coor_ contexts are superior to other dependency contexts including the set of all dependency contexts, although they are still inferior to _SPs_. Finally, there are substantially fewer _SP_ contexts compared to alternative representations, leading to a massive reduction in training time. On an 8G words corpus and a 32 core machine, the _SP_ model trains in 11 minutes, compared to 5 and 11 hours with _BOW_ and all dependency contexts, respectively.

## 1 Introduction

In recent years, vector space models (VSMs) have become prominent in NLP. VSMs are often evaluated by measuring their ability to predict human judgments of lexical semantic relations between pairs of words, mostly association or similarity. While many datasets for these tasks are limited to pairs of nouns, the recent SimLex999 word similarity dataset Hill et al. (2014) also consists of similarity scores for _verb_ and _adjective_ pairs. State-of-the-art VSMs such as word2vec skip-gram (_w2v-SG_, Mikolov et al. (2013a)) and GloVe Pennington et al. (2014) excel at noun-related tasks. However, their performance substantially decreases on _verb_ similarity prediction in SimLex999, and their adjective representations have rarely been evaluated (Section 2).

In this paper we show that a key factor in the reduced performance of the _w2v-SG_ model on verb representation is its reliance on bag-of-words (_BOW_) contexts: contexts of the represented words that consist of words in their physical proximity. We investigate a number of alternative contexts for this model, including various dependency contexts, and show that simple, automatically acquired symmetric patterns (_SPs_, e.g., "X or Y", Hearst (1992); Davidov and Rappoport (2006)) are the most useful contexts for the representation of verbs and also adjectives. Moreover, the _SP_-based model is much more compact than the alternatives, making its training an order of magnitude faster.

In particular, we train several versions of the _w2v-SG_ model, each with a different context type, and evaluate the resulting word embeddings on the task of predicting the similarity scores of the verb and adjective portions of SimLex999. Our results show that _SP_ contexts (SG-_SP_) obtain the best results on both tasks: Spearman's \(\rho\) scores of 0.459 on verbs and 0.651 on adjectives. These results are 15.2% and 4.7% better than _BOW_ contexts and 7.3% and 6.5% better than all dependency contexts (_DepAll_). Moreover, the number of _SP_ contexts is substantially

smaller than the alternatives, making it extremely fast to train: 11 minutes only on an 8G word corpus using a 32 CPU core machine, compared to 5 and 11 hours for _BOW_ and _DepAll_, respectively.

Recently, Schwartz et al. (2015) presented a count-based VSM that utilizes _SP_ contexts (SRR15). This model excels on verb similarity, outperforming VSMs that use other contexts (e.g., _BOW_ and _DepAll_) by more than 20%. In this paper we show that apart from its _SP_ contexts, the success of SRR15 is attributed in large to its explicit representation of antonyms (_live/die_); turning this feature off reduces its performance to be on par with SG-_SP_. As opposed to Schwartz et al. (2015), we keep our VSM fixed across experiments (_w2v-SG_), changing only the context type. This allows us to attribute our improved results to one factor: _SP_ contexts.

We further observe that _SP_ contexts are tightly connected to syntactic _coordination_ contexts (_Coor_, Section 3). Following this observation, we compare the _w2v-SG_ model with three dependency-based context types: (a) _Coor_ contexts; (b) all dependency links (_DepAll_); and (c) all dependency links excluding _Coor_ links (_Coor\({}^{C}\)_).1 Our results show that training with _Coor_ contexts is superior to training with the other context types, leading to improved similarity prediction of 2.7-4.1% and 4.3-6.9% on verbs and adjectives respectively.

Footnote 1: \(Coor\cup Coor^{C}=\textit{DepAll}\), \(Coor\cap Coor^{C}=\emptyset\)

These results demonstrate the prominence of _Coor_ contexts in verb and adjective representation: these contexts are even better than their combination with the rest of the dependency-based contexts (the _DepAll_ contexts). Nonetheless, although _Coor_ contexts are extracted using a supervised dependency parser, they are still inferior to _SP_ contexts, extracted automatically from plain text (Section 3), by 4.6% and 2.2% for verb and adjective pairs.

## 2 Background

Word Embeddings for Verbs and Adjectives.A number of evaluation sets consisting of word pairs scored by humans for semantic relations (mostly association and similarity) are in use for VSM evaluation. These include: RG-65 (Rubenstein and Goodenough, 1965), MC-30 (Miller and Charles, 1991), WordSim353 (Finkelstein et al., 2001), MEN (Bruni et al., 2014) and SimLex999 (Hill et al., 2014).2

Footnote 2: For a comprehensive list see: wordvectors.org/

Nouns are dominant in almost all of these datasets. For example, RG-65, MC-30 and WordSim353 consist of noun pairs almost exclusively. A few datasets contain pairs of verbs (Yang and Powers, 2006; Baker et al., 2014). The MEN dataset, although dominated by nouns, also contains verbs and adjectives. Nonetheless, the human judgment scores in these datasets reflect _relatedness_ between words. In contrast, the recent SimLex999 dataset (Hill et al., 2014) contains word _similarity_ scores for nouns (666 pairs), verbs (222 pairs) and adjectives (111 pairs). We use this dataset to study the effect of context type on VSM performance in a verb and adjective similarity prediction task.

Context Type in Word Embeddings.Most VSMs (e.g., Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014) define the context of a target word to be the words in its physical proximity (bag-of-words contexts). Dependency contexts, consisting of the words connected to the target word by dependency links (Grefenstette, 1994; Pado and Lapata, 2007; Levy and Goldberg, 2014), are another well researched alternative. These works did not recognize the importance of syntactic coordination contexts (_Coor_).

Patterns have also been suggested as VSMtexts, but mostly for representing _pairs_ of words (Turney, 2006; Turney, 2008). While this approach has been successful for extracting various types of word relations, using patterns to represent _single_ words is useful for downstream applications. Recently, Schwartz et al. (2015) explored the value of _symmetric_ pattern contexts for word representation, an idea this paper develops further.

A recently published approach (Melamud et al., 2016) also explored the effect of the type of context on the performance of word embedding models. Nonetheless, while they also explored bag-of-words and dependency contexts, they did not experiment with _SPs_ or coordination contexts, which we find to be most useful for predicting word similarity.

Limitations of Word Embeddings.Recently, a few papers examined the limitations of word embedding models in representing different types of se 

mantic information. Levy et al. (2015) showed that word embeddings do not capture semantic relations such as hyponymy and entailment. Rubinstein et al. (2015) showed that while state-of-the-art embeddings are successful at capturing taxonomic information (e.g., _cow_ is an animal), they are much less successful in capturing attributive properties (_bananas_ are yellow). In Schwartz et al. (2015), we showed that word embeddings are unable to distinguish between pairs of words with opposite meanings (antonyms, e.g., good/bad). In this paper we study the difficulties of bag-of-words based word embeddings in representing verb similarity.

## 3 Symmetric Patterns (_Sps_)

Lexico-syntactic patterns are templates of text that contain both words and wildcards Hearst (1992), e.g., "X _and_ Y" and "X _for a_ Y". Pattern _instances_ are sequences of words that match a given pattern, such that concrete words replace each of the wildcards. For example, "**John**_and_**Mary**" is an instance of the pattern "X _and_ Y". Patterns have been shown useful for a range of tasks, including word relation extraction Lin et al. (2003); Davidov et al. (2007), knowledge extraction Etzioni et al. (2005), sentiment analysis Davidov et al. (2010) and authorship distribution Schwartz et al. (2013).

_Symmetric_ patterns (_SPs_) are lexico-syntactic patterns that comply to two constraints: (a) Each pattern has exactly two wildcards (e.g., **X**_or_**Y**); and (b) When two words (_X,Y_) co-occur in an _SP_, they are also likely to co-occur in this pattern in opposite positions, given a large enough corpus (e.g., "**X**_or_**Y**" and "**Y**_or_**X**"). For example, the pattern "**X**_and_**Y**" is symmetric as for a large number of word pairs (e.g., \((eat,drink)\)) both members are likely to occur in both of its wildcard positions (e.g., "eat _and_ drink", "drink _and_ eat").

_SPs_ have shown useful for tasks such as word clustering Widdows and Dorow (2002); Davidov and Rappoport (2006), semantic class learning Kozareva et al. (2008) and word classification Schwartz et al. (2014). In this paper we demonstrate the value of _SP_-based contexts in vector representations of verbs and adjectives. The rationale behind this context type is that two words that co-occur in an _SP_ tend to take the same semantic role in the sentence, and are thus likely to be similar in meaning (e.g., "John and Mary) Sang").

_SP Extraction._ Many works that applied _SPs_ in NLP tasks employed a hand-crafted list of patterns Wildows and Dorow (2002); Dorow et al. (2005); Feng et al. (2013). Following Schwartz et al. (2015) we employ the dr06 algorithm Davidov and Rappoport (2006), an unsupervised algorithm that extracts _SPs_ from plain text. We apply this algorithm to our corpus (Section 4) and extract 11 _SPs_: "X _and_ Y", "X _or_ Y", "X _and the_ Y", "X _or_ the_ Y", "X _or_ a_ Y", "X _nor_ Y", "X _and one_ Y", "either_ X _or_ Y", "X rather than_ Y", "X _as well as_ Y", "_from_ X _to_ Y". A description of the dr06 algorithm is beyond the scope of this paper; the interested reader is referred to Davidson and Rappoport (2006).

_SP_ **Contexts.** We generate _SP_ contexts by taking the co-occurrence counts of pairs of words in _SPs_. For example, in the _SP_ token "_boys__and girls_", the term _girls_ is taken as an _SP_ context of the word _boys_, and _boys_ is taken as an _SP_ context of _girls_.

We do not make a distinction between the different _SPs_. E.g., "_boys_**and**_girls_" and "_boys_**or**_girls_" are treated the same. However, we distinguish between left and right contexts. For example, we generate different contexts for the word _girls_, one for left-hand contexts ("**girls**_and boys_") and another for right-hand contexts ("_boys_and**girls_").

_SPs_ and **Coordinations.**_SPs_ and syntactic coordinations (_Coors_) are intimately related. For example, of the 11 _SPs_ extracted in this paper by the dr06 algorithm (listed above), the first eight represent coordination structures. Moreover, these _SPs_ account for more than 98% of the _SP_ instances in our corpus. Indeed, due to the significant overlap between _SPs_ and _Coors_, the former have been proposed as a simple model of the latter Nakov and Hearst (2005).3

Footnote 3: Note though that the exact syntactic annotation of coordination is debatable both in the linguistic community Tesniere (1959); Hudson (1980); Mel’cuk (1988) and also in the NLP community Nilsson et al. (2006); Schwartz et al. (2011); Schwartz et al. (2012).

Despite their tight connection, _SPs_ sometimes fail to properly identify the components of _Coors_. For example, while _SPs_ are instrumental in capturing shallow _Coors_, they fail in capturing coordination between phrases. Consider the sentence _John 

_walked and Mary ran_: the _SP_ "X _and_ Y" captures the phrase _walked and Mary_, while the _Coor_ links the heads of the connected phrases ("_walked_" and "_ran_"). _Sps_, on the other hand, can go beyond _Coors_ and capture other types of symmetric structures like "_from X to Y_" and "X _rather than_ Y".

Our experiments reveal that both _SPs_ and _Coors_ are highly useful contexts for verb and adjective representation, at least with respect to word similarity. Interestingly, _Coor_ contexts, extracted using a supervised dependency parser, are less effective than _SP_ contexts, which are extracted from plain text.

## 4 Experiments

Model.We keep the VSM fixed throughout our experiments, changing only the context type. This methodology allows us to evaluate the impact of different contexts on the VSM performance, as context choice is the only modeling decision that changes across experimental conditions.

Our VSM is the word2vec skip-gram model (_w2v-SG_, Mikolov et al. Mikolov13a), which obtains state-of-the-art results on a variety of NLP tasks Baroni et al. (2014). We employ the word2vec toolkit.5 For all context types other than _BOW_ we use the word2vec package of Levy and Goldberg (2014),6 which augments the standard word2vec toolkit with code that allows arbitrary context definition.

Footnote 5: [https://code.google.com/p/word2vec/](https://code.google.com/p/word2vec/)

Experimental Setup.We experiment with the verb pair (222 pairs) and adjective pair (111 pairs) portions of SimLex99 (Hill et al., 2014). We report the Spearman \(\rho\) correlation between the ranks derived from the scores of the evaluated models and the human scores provided in SimLex99.7

Footnote 7: [https://bitbucket.org/yoavgo/word2vecf](https://bitbucket.org/yoavgo/word2vecf)

Footnote 8: Model scores are computed in the standard way: applying the cosine similarity metric to the vectors learned for the words participating in the pair.

We train the _w2v-SG_ model with five different context types: (a) _BOW_ contexts (SG-_BOW_); (b) all dependency links (SG-_DepAll_) (c) dependency-based coordination contexts (i.e., those labeled with _conj_, SG-_Coor_); (d) all dependency links except for coordinations (SG-_Coor\({}^{C}\)_); and (e) _SP_ contexts. Our training corpus is the 8G words corpus generated by the word2vec script.9

Footnote 9: Code.google.com/p/word2vec/source/browse/trunk/demo-train-big-model-vl.sh

Models (b)-(d) require the dependency parse trees of the corpus as input. To generate these trees, we employ the Stanford POS Tagger Toutanova et al. (2003)10 and the stack version of the MALT parser Nivre et al. (2009).11 The _SP_ contexts are generated using the _SPs_ extracted by the dr06 algorithm from our training corpus (see Section 3).

Footnote 11: [http://www.maltparser.org/index.html](http://www.maltparser.org/index.html)

For _BOW_ contexts, we experiment with three window sizes (2, 5 and 10) and report the best results (window size of 2 across conditions). For dependency based contexts we follow the standard convention in the literature: we consider the immediate heads and modifiers of the represented word. All models are trained with 500 dimensions, the default value of the word2vec script. Other hyper-parameters were also set to the default values of the code packages.

Results.Table 1 presents our results. The SG-_SP_ model provides the most useful verb and adjective representations among the _w2v-SG_ models. Compared to _BOW_ (SG-_BOW_), the most commonly used

\begin{table}
\begin{tabular}{|l|r|r|r||r|} \hline
**Model** & **Verb** & **Adj.** & **Noun** & **Time** & **\#Cont.** \\ \hline SG-_BOW_ & 0.307 & 0.604 & **0.501** & 320 & 13G \\ \hline SG-_DepAll_ & 0.386 & 0.586 & 0.499 & 551 & 14.5G \\ \hline SG-_Coor_ & 0.413 & 0.629 & 0.428 & 23 & 550M \\ \hline SG-_Coor\({}^{C}\)_ & 0.372 & 0.56 & 0.494 & 677 & 14G \\ \hline SG-_SP_ & **0.459** & **0.651** & 0.415 & 11 & 270M \\ \hline \hline SRR15 & 0.578 & 0.663 & 0.497 & — & 270M \\ \hline SRR15\({}^{-}\) & 0.441 & 0.68 & 0.421 & — & 270M \\ \hline \end{tabular}
\end{table}
Table 1: 

context type, SG-_SP_ results are 15.2% and 4.7% higher on verbs and adjectives respectively. Compared to dependency links (SG-_DepAll_), the improvements are 7.3% and 6.5%. For completeness, we compare the models on the noun pairs portion, observing that SG-_BOW_ and SG-_DepAll_ are \({\sim}8.5\%\) better than SG-_SP_. This indicates that different word classes require different representations.

The results for SG-_Coor_, which is trained with syntactic coordination (_Coor_) contexts, show that these contexts are superior to all the other dependency links (SG-_Coor\({}^{C}\)_) by 4.1% and 6.9% on verbs and adjectives. Importantly, comparing the SG-_Coor_ model to the SG-_DepAll_ model, which augments the _Coor_ contexts with the other syntactic dependency contexts, reveals that SG-_DepAll_ is actually inferior by 2.7% and 4.3% in Spearman \(\rho\) on verbs and adjectives respectively. Interestingly, _Coor_ contexts, which are extracted using a supervised parser, are still inferior by 4.6% and 2.2% to _SPs_, which capture similar contexts but are extracted from plain text.

Table 1 also shows the training times of the various _w2v-SG_ models on a 32G memory, 32 CPU core machine. SG-_SP_ and SG-_Coor_, which take 11 minutes and 23 minutes respectively to train, are substantially faster than the other _w2v-SG_ models. For example, they are more than an order of magnitude faster than SG-_BOW_ (320 minutes) and SG-_Coor\({}^{C}\)_ (677 minutes). This is not surprising, as there are far fewer _SP_ contexts (270M) and _Coor_ contexts (550M) than _BOW_ contexts (13G) and _Coor\({}^{C}\)_ contexts (14G) (#Cont. column).

Finally, the performance of the SG-_SP_ model is still substantially inferior to the SRR15 _SP_-based model Schwartz et al. (2015). As both models use the same _SP_ contexts, this result indicates that other modeling decisions in SRR15 lead to its superior performance. We show that this difference is mostly attributed to one feature of SRR15: its method for detecting antonym pairs (_good/bad_). Indeed, the SRR15 model without its antonym detection method (SRR15\({}^{-}\)) obtains a Spearman \(\rho\) of 0.441, compared to 0.459 of SG-_SP_ on verb pairs. For adjectives, however, SRR15\({}^{-}\) is 1.7% better than SRR15, increasing the difference from SG-_SP_ to 2.9%.11

Footnote 10: We compare the _w2v-SG_ models training time only. SRR15 and SRR15\({}^{-}\) are count-based models and have no training step.

## 5 Conclusions

We demonstrated the effectiveness of symmetric pattern contexts in word embedding induction. Experiments with the word2vec model showed that these contexts are superior to various alternatives for verb and adjective representation. We further pointed at the connection between symmetric patterns and syntactic coordinations. We showed that coordinations are superior to other syntactic contexts, but are still inferior to symmetric patterns, although the extraction of symmetric patterns requires less supervision.

Future work includes developing a model that successfully combines the various context types explored in this paper. We are also interested in the representation of other word classes such as adverbs for which no evaluation set currently exists. Finally, the code for generating the SG-_SP_ embeddings, as well as the vectors experimented with in this paper, are released and can be downloaded from [http://www.cs.huji.ac.il/~roys02/papers/sp_sg/sp_sg.html](http://www.cs.huji.ac.il/~roys02/papers/sp_sg/sp_sg.html)

## Acknowledgments

This research was funded (in part) by the Intel Collaborative Research Institute for Computational Intelligence (ICRI-CI), the Israel Ministry of Science and Technology Center of Knowledge in Machine Learning and Artificial Intelligence (Grant number 3-9243). The second author was partially funded by the Microsoft/Technion research center for electronic commerce and the Google faculty research award.

## References

* Baker et al. [2014] Simon Baker, Roi Reichart, and Anna Korhonen. 2014. An unsupervised model for instance level subcategorization acquisition. In _Proc. of EMNLP_.
* Baroni et al. [2014] Marco Baroni, Georgiana Dinu, and German Kruszewski. 2014. Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In _Proc. of ACL_.

 

Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. _JAIR_.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _JMLR_, 12:2493-2537.
* Davidov and Rappoport (2006) Dmitry Davidov and Ari Rappoport. 2006. Efficient unsupervised discovery of word categories using symmetric patterns and high frequency words. In _Proc. of ACL-COLING_.
* Davidov et al. (2007) Dmitry Davidov, Ari Rappoport, and Moshe Koppel. 2007. Fully unsupervised discovery of concept-specific relationships by web mining. In _Proc. of ACL_.
* Davidov et al. (2010) Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Enhanced sentiment learning using twitter hashtags and smileys. In _Proc. of COLING_.
* Dorow et al. (2005) Beate Dorow, Dominic Widdows, Katarina Ling, Jean-Pierre Eckmann, Danilo Sergi, and Elisha Moses. 2005. Using Curvature and Markov Clustering in Graphs for Lexical Acquisition and Word Sense Discrimination.
* Etzioni et al. (2005) Oren Etzioni, Michael Cafarella, Doug Downey, Ana-Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S Weld, and Alexander Yates. 2005. Unsupervised named-entity extraction from the web: An experimental study. _Artificial intelligence_, 165(1):91-134.
* Feng et al. (2013) Song Feng, Jun Seok Kang, Polina Kuznetsova, and Yejin Choi. 2013. Connotation lexicon: A dash of sentiment beneath the surface meaning. In _Proc. of ACL_.
* Finkelstein et al. (2001) Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In _Proc. of WWW_.
* Grefenstette (1994) Gregory Greenstette. 1994. _Explorations in automatic thesaurus discovery_. Boston: Kluwer.
* Volume 2_.
* Hill et al. (2014) Felix Hill, Roi Reichart, and Anna Korhonen. 2014. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. _arXiv:1408.3456 [cs.CL]_.
* Hudson (1980) Richard A. Hudson. 1980. _Arguments for a Non-transformational Grammar_. Chicago: University of Chicago Press.
* Kozareva et al. (2008) Zornitsa Kozareva, Ellen Riloff, and Eduard Hovy. 2008. Semantic class learning from the web with hyponym pattern linkage graphs. In _Proc. of ACL-HLT_.
* Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In _Proc. of ACL_.
* Levy et al. (2015) Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? In _Proc. of NAACL_.
* Lin et al. (2003) Dekang Lin, Shaojun Zhao, Lijuan Qin, and Ming Zhou. 2003. Identifying synonyms among distributionally similar words. In _Proc. of IJCAI_.
* Melamud et al. (2016) Oren Melamud, David McClosky, Siddharth Patwardhan, and Mohit Bansal. 2016. The role of context types and dimensionality in learning word embeddings. In _Proc. of NAACL_.
* Mel'cuk (1988) Igor Mel'cuk. 1988. _Dependency Syntax: Theory and Practice_. State University of New York Press.
* Mikolov et al. (2013a) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013a. Distributed representations of words and phrases and their compositionality. In _Proc. of NIPS_.
* Mikolov et al. (2013b) Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In _Proc. of NAACL-HLT_.
* Miller and Charles (1991) George A Miller and Walter G Charles. 1991. Contextual correlates of semantic similarity. _Language and cognitive processes_.
* Nakov and Hearst (2005) Preslav Nakov and Marti Hearst. 2005. Using the web as an implicit training set: application to structural ambiguity resolution. In _Proc. of HLT-EMNLP_.
* Nilsson et al. (2006) Jens Nilsson, Joakim Nivre, and Johan Hall. 2006. Graph transformations in data-driven dependency parsing. In _Proc. of ACL-COLING_.
* Nivre et al. (2009) Joakim Nivre, Marco Kuhlmann, and Johan Hall. 2009. An improved oracle for dependency parsing with online reordering. In _Proc. of IWPT_.
* Pado and Lapata (2007) Sebastian Pado and Mirella Lapata. 2007. Dependency-based construction of semantic space models. _Computational Linguistics_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Proc. of EMNLP_.
* Rubenstein and Goodenough (1965) Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. _Communications of the ACM_.
* Rubinstein et al. (2015) Dana Rubinstein, Effi Levi, Roy Schwartz, and Ari Rappoport. 2015. How well do distributional models capture different types of semantic knowledge? In _Proc. of ACL_.
* Schwartz et al. (2011) Roy Schwartz, Omri Abend, Roi Reichart, and Ari Rappoport. 2011. Neutralizing linguistically problematic annotations in unsupervised dependency parsing evaluation. In _Proc. of ACL-HLT_.
* Schwartz et al. (2012) Roy Schwartz, Omri Abend, and Ari Rappoport. 2012. Learnability-based syntactic annotation design. In _Proc. of COLING_.
* Schwartz et al. (2013) Roy Schwartz, Oren Tsur, Ari Rappoport, and Moshe Koppel. 2013. Authorship attribution of micromessages. In _Proc. of EMNLP_.

 

Roy Schwartz, Roi Reichart, and Ari Rappoport. 2014. Minimally supervised classification to semantic categories using automatically acquired symmetric patterns. In _Proc. of COLING_.
* Schwartz et al. (2015) Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015. Symmetric pattern based word embeddings for improved word similarity prediction. In _Proc. of CoNLL_.
* Tensiere (1959) Lucien Tesniere. 1959. _Elements de syntaxe structurale_. Paris: Klincksieck.
* Toutanova et al. (2003) Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In _Proc. of NAACL_.
* Turney (2006) Peter D. Turney. 2006. Similarity of semantic relations. _Computational Linguistics_.
* Turney (2008) Peter D. Turney. 2008. A uniform approach to analogies, synonyms, antonyms, and associations. In _Proc. of COLING_.
* Widdows and Dorow (2002) Dominic Widdows and Beate Dorow. 2002. A graph model for unsupervised lexical acquisition. In _Proc. of COLING_.
* Yang and Powers (2006) Dongqiang Yang and David M. W. Powers. 2006. Verb similarity on the taxonomy of wordnet. In _Proc. of GWC_.