Venue: NAACL 2018

# Key2Vec: Automatic Ranked Keyphrase Extraction from Scientific

Articles using Phrase Embeddings

Debanjan Mahata Bloomberg New York, U.S.A dmahata@bloomberg.net &John Kuriakose Infosys Limited Pune, India John_Kuriakose@infosys.com &Rajiv Ratn Shah IIIT-Delhi New Delhi, India rajivratn@iiitd.ac.in &Roger Zimmermann NUS-Singapore, rogerz@comp.nus.edu.sg

###### Abstract

Keyphrase extraction is a fundamental task in natural language processing that facilitates mapping of documents to a set of representative phrases. In this paper, we present an unsupervised technique (_Key2Vec_) that leverages phrase embeddings for ranking keyphrases extracted from scientific articles. Specifically, we propose an effective way of processing text documents for training multi-word phrase embeddings that are used for thematic representation of scientific articles and ranking of keyphrases extracted from them using theme-weighted PageRank. Evaluations are performed on benchmark datasets producing state-of-the-art results.

## 1 Introduction and Background

Keyphrases are single or multi-word linguistic units that represent the salient aspects of a document. The task of ranked keyphrase extraction from scientific articles is of great interest to scientific publishers as it helps to recommend articles to readers, highlight missing citations to authors, identify potential reviewers for submissions, and analyze research trends over time Augenstein et al. (2017). Due to its widespread use, keyphrase extraction has received significant attention from researchers Kim et al. (2010); Augenstein et al. (2017). However, the task is far from solved and the performances of the present systems are worse in comparison to many other NLP tasks Liu et al. (2010). Some of the major challenges are the varied length of the documents to be processed, their structural inconsistency and developing strategies that can perform well in different domains Hasan and Ng (2014).

Methods for automatic keyphrase extraction are mainly divided into two categories: _supervised_ and _unsupervised_. Supervised methods approach the problem as a binary classification problem Hasan and Ng (2014), whereas the unsupervised methods are mostly based on TF-IDF, clustering, and graph-based ranking Hasan and Ng (2010); Mihalcea and Tarau (2004). On the presence of domain-specific data, supervised methods have shown better performance. The unsupervised methods have the advantage of not requiring any training data and can produce results in any domain.

With recent advancements in deep learning techniques applied to natural language processing (NLP), the trend is to represent words as dense real-valued vectors, popularly known as word embeddings. These representations of words have been shown to equal or outperform other methods (e.g. LSA, SVD) Baroni et al. (2014). The embedding vectors, are supposed to preserve the semantic and syntactic similarities between words. They have been shown to be useful for several NLP tasks, like part-of-speech tagging, chunking, named entity recognition, semantic role labeling, syntactic parsing, and speech processing, among others Collobert et al. (2011). Some of the most popular approaches for training word embeddings are Word2Vec Mikolov et al. (2013), Glove Pennington et al. (2014) and Fasttext Bojanowski et al. (2016).

Word embeddings have already shown promis

\begin{table}
\begin{tabular}{c c}  & **Title: Identification of states of complex systems with estimation of admissible measurement errors on the basis of fuzzy information.** \\ \hline \end{tabular}
\end{table}
Table 1: Keyphrases extracted by using _Key2Vec_ from a sample research article abstract.



ing results in the process of keyphrase extraction from scientific articles (Wang et al., 2015, 2014). However, Wang _et al._ did not use domain-specific word embeddings and had suggested that training them might lead to improvements. This motivated us to experiment with domain-specific embeddings on scientific articles.

In this work, we represent candidate keyphrases extracted from a scientific article by domain-specific phrase embeddings and rank them using a _theme-weighted_ PageRank algorithm (Langville and Meyer, 2004), such that the _thematic weight_ of a candidate keyphrase indicate how similar it is to the _thematic representation_ or the main theme of the article, which is also constructed using the same embeddings. Due to extensive use of phrase embeddings for representing the candidate keyphrases and ranking them, we name our method as _Key2Vec_. To our knowledge, using multi-word phrase embeddings for constructing _thematic representation_ of a given document and to assign _thematic weights_ to phrases have not been used for ranked keyphrase extraction, and this work is the first preliminary attempt to do so. Table 1. shows ranked keyphrases extracted using _Key2Vec_ from a sample research abstract. Next, we present our methodology.

## 2 Methodology

Our methodology primarily uses three steps: _candidate selection_, _candidate scoring_, and _candidate ranking_, similar to other popular frameworks of ranked keyphrase extraction (Kim et al., 2013). All the steps depend on the choice of our text processing steps and a phrase embedding model that we train on a large corpus of scientific articles. We explain them next and give a detailed description of their implementations.

### Text Processing

It has been shown (Mikolov et al., 2013), that the presence of multi-word phrases intermixed with unigram words increases the performance and accuracy of the embedding models trained using techniques such as _Word2Vec_. However, in our framework we take a different approach in detecting meaningful and cohesive chunk of phrases while preparing the text samples for training. Instead of relying on measures considering how often two or more words co-occur with each other, we rely on already trained dependency parsing and named entity extraction models. For this work we use _Spacy_1 as our NLP toolkit along with its default models. The choice of _Spacey_ is just for convenience and is not driven by any other factor. We split a text document into sentences, tokenize a sentence into unigram tokens, as well as identify noun phrases and named entities from it. During this process if a named entity is detected at a particular offset in the sentence then a noun phrase appearing at the same offset is not considered.

Footnote 1: [https://spacy.io](https://spacy.io)

We take steps in cleaning the individual single word and multi-word tokens that we obtain. Specifically, we filter out the following tokens.

* Noun phrases and named entities that are fully numeric.
* Named entities that belong to the following categories are filtered out : DATE, TIME, PERCENT, MONEY, QUANTITY, ORDINAL, CARDINAL. Refer, Spacy's named entity documentation2 for details of the tags. Footnote 2: [https://spacy.io/usage/linguistic-features#section-named-entities](https://spacy.io/usage/linguistic-features#section-named-entities)
* Standard stopwords are removed.
* Punctuations are removed except '-'.

We also take steps to clean leading and ending tokens of a multi-word noun phrase and named entity.

* Common adjectives and reporting verbs are removed if they occur as the first or last token of a noun phrase/named entity.
* Determiners are removed from the first token of a noun phrase/named entity.
* First or last tokens of noun phrases/named entities belonging to following parts of

Figure 1: Text processing pipeline for preparing text samples used for training word embedding models.

 

speech: INTJ Interjection, AUX Auxiliary, CCONJ Coordinating Conjunction, ADP Adposition, DET Interjection, NUM Numeral, PART Particle, PRON Pronoun, SCONJ Subordinating Conjunction, PUNCT Punctutation, SYM Symbol, X Other, are removed. For a detailed reference of each of these POS tags please refer Spacy's documentation3. Footnote 2: [https://spacy.io/api/annotation#pos-tagging](https://spacy.io/api/annotation#pos-tagging)
* Starting and ending tokens of a noun phrase/named entity is removed if they belong to a standard list of english stopwords and functional words.

Apart from relying on Spacy's parser we use hand crafted regexes for cleaning the final list of tokens obtained after the above data cleaning steps.

* Get rid of leading/trailing junk characters.
* Handle dangling/backwards parentheses. We don't allow '(' or ')' to appear without the other.
* Handle oddly separated hyphenated words.
* Handle oddly separated apostrophe'd words.
* Normalize whitespace.

The resultant unigram tokens and multi-word phrases are merged in the order they appeared in the original sentence. Figure 1, shows an example of how the text processing pipeline works on an example sentence for preparing the training samples that act as an input to the embedding algorithm.

### Training Phrase Embedding Model

The methodology to a great extent relies on the underlying embeddings. We directly train multi-word phrase embeddings using _Fasttext4_, rather than first training embedding models for unigram words and then combining their dense vectors to obtain vectors for multi-word phrases. Our training vocabulary consists of both unigram as well as multi-word phrases. We are aware of the existing procedures for training phrase embeddings [60, 58], but refrain from using them in this preliminary work. We would like to use them in the future.

Footnote 4: [https://fasttext.cc/](https://fasttext.cc/)

The main aim of the underlying embedding model is to capture semantic and syntactic similarities between textual units comprising of both single word and multi-word phrases. We chose _Fasttext_ over other embedding techniques because it captures both the semantic and morphological similarities5 between words. For example, if we have _breast cancer_ in the content of the theme of a document then intuitively phrases like _breast cancer_, _breast cancer treatment_, should be assigned higher thematic weight than _prostrate cancer_ or _lung cancer_, even though the document might mention other forms of cancer as well. Embedding techniques like _WordZVec_ and _Glove_, only takes into account the semantic similarity between words based on their occurrences in a similar context and will not display the desired property that we want to leverage. We would like to study the effects of other types of embeddings in the future.

Footnote 5: [https://rare-technologies.com/fasttext-and-gensim-word-embeddings/](https://rare-technologies.com/fasttext-and-gensim-word-embeddings/)

**Dataset:** Since this work deals with the domain of scientific articles we train the embedding model on a collection of more than million scientific documents. We collect 1,147,000 scientific abstracts related to different areas (Fig 2) from arxiv.org6. For collecting data we use the API provided by arxiv.org that allows bulk access7 to the articles uploaded in their portal. We also add the scientific documents present in the benchmark datasets (Sections 3), increasing the total number of documents to 1,149,244.

Footnote 6: [http://arxiv.org](http://arxiv.org)

After processing the text of the documents as mentioned above, we train a _Fasttext-skipgram_ model using _negative sampling_ with a _context win

Figure 2: Frequency distribution of topics in the arxiv dataset used for training phrase embeddings.

 

_down size_ of \(5\), _dimension_ of \(100\) and number of _epochs_ set to \(10\). We would like to experiment further in the future on the selection of optimal parameters for the embedding model.

**Candidate Selection:** This step aids in chosing candidate keyphrases from the set of all phrases that can be extracted from a document, and is commonly used in most of the automated ranked keyphrase extraction systems. Not all the phrases are considered as candidates. Generally, unwanted and noisy phrases are eliminated in this process by using different heuristics (Section 2.1). We split a given document into sentences and to extract noun phrases and named entities as described previously. As an output of this step we get a set of unique phrases (\(C_{d_{i}}=\{c_{1},c_{2},...,c_{n}\}_{d_{i}}\)) for a document \(d_{i}\) to be used later for scoring and ranking in the next two steps.

**Candidate Scoring:** In this step we assign a _theme vector_ (\(\hat{r_{d_{i}}}\)) to a document (\(d_{i}\)). The _theme vector_ can be tuned according to the type of documents that are being processed and the type of keyphrases that we want to get in our final results. In this work, we extract a _theme excerpt_ from a given document and further extract a unique set of _thematic phrases_ comprising of named entities, noun phrases and unigram words (\(T_{d_{i}}=\{t_{1},t_{2},...,t_{m}\}_{d_{i}}\)) from it. For the _Inspec_ dataset we use the first sentence of the document which consists its title, and for the _SemEval_ dataset we use the title and the first ten sentences extracted from the beginning of the document, as the _theme excerpts_, respectively (see Section 3). The first ten sentences of a document from the _SemEval_ dataset essentially captures the abstract and sometimes first few sentences of the introduction of a scientific article. We get the vector representation (\(\hat{t_{j}}\)) of each _thematic phrase_ extracted from the _theme excerpt_ using the phrase embedding model that we trained and perform vector addition in order to get the final _theme vector_ (\(\hat{r_{d_{i}}}=\sum_{j=1}^{m}\hat{t_{j}}\)) of the document. The phrase embedding model is then used to get the vector representation (\(\hat{c_{k}};k\in\{1...n\}\)) for each candidate keyphrase in \(C_{d_{i}}\).

We calculate the _cosine distance_ between the _theme vector_ (\(\hat{r_{d_{i}}}\)) and vector for each candidate keyphrase (\(\hat{c_{k}}\)) and assign a score (\(\kappa(\hat{x},\hat{y})\rightarrow[0,1]\)) to each candidate, with \(1\) indicating a complete similarity with the _theme vector_ and \(0\) indicating a complete dissimilarity. To get the final _thematic weight_ (\(w_{c_{j}}^{d_{i}}\)) for each candidate w.r.t. a given document (\(d_{i}\)), the candidate scores are scaled again between \(0\) and \(1\) with a score of \(1\) assigned to the candidate semantically closest to the main theme of the document and \(0\) to the farthest.

**Candidate Ranking:** In order to perform final ranking of the candidate keyphrases we use weighted personalized PageRank algorithm. A directed graph \(G_{d_{i}}\) is constructed for a given document (\(d_{i}\)) with \(C_{d_{i}}\) as the vertices and \(E_{d_{i}}\) as the edges connecting two candidate keyphrases if they co-occur within a window size of \(5\). The edges are bidirectional. Weights \(sr(c_{j}^{d_{i}},c_{k}^{d_{i}})\) are calculated for the edges using the semantic similarity between the candidate kephrases obtained from the phrase embedding model and their frequency of co-occurrence, as used by Wang _et al._Wang et al. (2015), and shown in equation 3. We use _cosine distance_ (\(\frac{1}{1-cosine(c_{j}^{d_{i}},c_{k}^{d_{i}})}\)) and _Point-wise Mutual Information_ (\(PMI(c_{j}^{d_{i}},c_{k}^{d_{i}})\)) for calculating \(semantic(c_{j}^{d_{i}},c_{k}^{d_{i}})\) (equation 1) and \(cooccur(c_{j}^{d_{i}},c_{k}^{d_{i}})\) (equation 2), respectively. The main intuition behind calculating semantic relatedness by using a phrase embedding model is to capture how well two phrases are related to each other in general. Whereas, the co-occurrence score captures the local relationship between the phrases within the context of the given document.

\[semantic(c_{j}^{d_{i}},c_{k}^{d_{i}})=\frac{1}{1-cosine(c_{j}^{d_{i}},c_{k}^{ d_{i}})} \tag{1}\]

\[cooccur(c_{j}^{d_{i}},c_{k}^{d_{i}})=PMI(c_{j}^{d_{i}},c_{k}^{d_{i}}) \tag{2}\]

\[sr(c_{j}^{d_{i}},c_{k}^{d_{i}})=semantic(c_{j}^{d_{i}},c_{k}^{d_{i}})\times cooccur (c_{j}^{d_{i}},c_{k}^{d_{i}}) \tag{3}\]

Given graph \(G\), if \(\varepsilon(c_{j}^{d_{i}})\) be the set of all edges incident on the vertex \(c_{j}^{d_{i}}\), and \(w_{c_{j}}^{d_{i}}\) is the _thematic weight_ of \(c_{j}^{d_{i}}\) as calculated in the _candidate scorning_ step, then the final PageRank score \(R(c_{j}^{d_{i}})\) of a candidate keyphrase \(c_{j}^{d_{i}}\) is calculated using equation 4, where \(d=0.85\) is the _lamping factor_ and \(out(c_{k}^{d_{i}})\) is the out-degree of the vertex \(c_{k}^{d_{i}}\).

\[R(c_{j}^{d_{i}})=(1-d)w_{c_{j}}^{d_{i}}+d\times\sum_{c_{k}^{d_{i}}\in(c_{j}^{d _{i}})}(\frac{sr(c_{j}^{d_{i}},c_{k}^{d_{i}})}{|out(c_{k}^{d_{i}})|})R(c_{k}^{ d_{i}}) \tag{4}\]

Next, we evaluate the performance of _KeyZVec_.

 



## 3 Experiments and Results

The final ranked keyphrases obtained using the _Key2Vec_ methodology as described in the previous section is evaluated on the popular _Inspec_ and _SemEval 2010_ datasets. The Inspec dataset [8] is composed of 2000 abstracts of scientific articles divided into sets of 1000, 500, and 500, as training, validation and test datasets respectively. Each document has two lists of keyphrases assigned by humans - _controlled_, which are assigned by the authors, and _uncontrolled_, which are freely assigned by the readers. The controlled keyphrases are mostly abstractive, whereas the uncontrolled ones are mostly extractive [46]. The _Semeval 2010_ dataset [9] consists of 284 full length ACM articles divided into a test set of size 100, training set of size 144 and trial set of size 40. Each article has two sets of human assigned keyphrases: the _authorassigned_ and _reader-assigned_ ones, equivalent to the _controlled_ and _uncontrolled_ categories, respectively of the _Inspec_ dataset. We only use the test datasets for our evaluations and combine the annotated _controlled_ and _uncontrolled_ keyphrases.

The ranked keyphrases are evaluated using exact match evaluation metric as used in SemEval 2010 Task 5. We match the keyphrases in the annotated documents in the benchmark datasets with those generated by _Key2Vec_, and calculate microaveraged precision, recall and F-score (\(\beta=1\)), respectively. In the evaluation, we check the performance over the top 5, 10 and 15 candidates returned by _Key2Vec_. The performance of _Key2Vec_ on the metrics is shown in Table 2. Tables 3 and 4 shows a comparison of _Key2Vec_ with some of the state-of-the-art systems giving best performances on the _Inspec_ and _SemEval 2010_ datasets, respectively.

## 4 Conclusion and Future Work

In this paper, we proposed a framework for automatic extraction and ranking of keyphrases from scientific articles. We showed an efficient way of training phrase embeddings, and showed its effectiveness in constructing thematic representation of scientific articles and assigning thematic weights to candidate keyphrases. We also introduced theme-weighted PageRank to rank the candidate keyphrases. Experimental evaluations confirm that our proposed technique of _Key2Vec_ produces state-of-the-art results on benchmark datasets. In the future, we plan to use other existing procedures for training phrase embeddings and study their effects. We also plan to use _Key2Vec_ in other domains such as news articles and extend the methodology for other related tasks like summarization.

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline
**SemEval 2010** & **Key2Vec** & \begin{tabular}{c} **SGRank** \\ **(Danesh et al.,** \\ **2015)** \\ \end{tabular} & \begin{tabular}{c} **HUMB** \\ **(Lopez and Romary,** \\ **2010)** \\ \end{tabular} & 
\begin{tabular}{c} **TopicRank** \\ **(Bougouin et al.,** \\ **2013)** \\ \end{tabular} \\ \hline
**Micro Avg. F1@10** & **29.04 \%** & 26.07 \% & 22.50 \% & 12.1 \% \\ \hline \end{tabular}
\end{table}
Table 4: Comparison of _Key2Vec_ with some state-of-the-art systems [10, 11, 12, 13] for Avg. F1@10 on _SemEval 2010_ dataset.

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|} \hline  & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Micro** \\ **Avg.** \\ **Precision** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Micro** \\ **Avg.** \\ **Recall** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Micro** \\ **Avg.** \\ **F1** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Micro** \\ **Avg.** \\ **Precision** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Micro** \\ **Avg.** \\ **Precision** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Micro** \\ **Avg.** \\ **Precision** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Micro** \\ **Avg.** \\ **Precision** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Micro** \\ **Avg.** \\ **Precision** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Argo** \\ **Avg.** \\ **Precision** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Arg.** \\ **Accuracy** \\ **F1** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Argo** \\ **Precision** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{\begin{tabular}{c} **Argo** \\ **Avg.** \\ **Precision** \\ \end{tabular} }} & \multicolumn{1}{l|}{\multirow{2}{*}{
\begin{tabular}{c} **Argo** \\ **Avg.





## References

* Augenstein et al. (2017) Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. Semeval 2017 task 10: Science-extracting keyphrases and relations from scientific publications. _arXiv preprint arXiv:1704.02853_ .
* Baroni et al. (2014) Marco Baroni, Georgiana Dinu, and German Kruszewski. 2014. Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In _ACL (1)_. pages 238-247.
* Bojanowski et al. (2016) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. _arXiv preprint arXiv:1607.04606_ .
* Boughouin et al. (2013) Adrien Boughouin, Florian Boudin, and Beatrice Daille. 2013. Topicrank: Graph-based topic ranking for keyphrase extraction. In _International Joint Conference on Natural Language Processing (IJCNLP)_. pages 543-551.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_ 12(Aug):2493-2537.
* Danesh et al. (2015) Soheil Danesh, Tamara Sumner, and James H Martin. 2015. Sgrank: Combining statistical and graphical methods to improve the state of the art in unsupervised keyphrase extraction. _Lexical and Computational Semantics (* SEM 2015)_ page 117.
* Hasan and Ng (2010) Kazi Saidul Hasan and Vincent Ng. 2010. Conundrums in unsupervised keyphrase extraction: making sense of the state-of-the-art. In _Proceedings of the 23rd International Conference on Computational Linguistics: Posters_. Association for Computational Linguistics, pages 365-373.
* Hasan and Ng (2014) Kazi Saidul Hasan and Vincent Ng. 2014. Automatic keyphrase extraction: A survey of the state of the art. In _ACL (1)_. pages 1262-1273.
* Huth (2003) Anette Huth. 2003. Improved automatic keyword extraction given more linguistic knowledge. In _Proceedings of the 2003 conference on Empirical methods in natural language processing_. Association for Computational Linguistics, pages 216-223.
* Kim et al. (2010) Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2010. Semeval-2010 task 5: Automatic keyphrase extraction from scientific articles. In _Proceedings of the 5th International Workshop on Semantic Evaluation_. Association for Computational Linguistics, pages 21-26.
* Kim et al. (2013) Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2013. Automatic keyphrase extraction from scientific articles. _Language resources and evaluation_ 47(3):723-742.
* Langville and Meyer (2004) Amy N Langville and Carl D Meyer. 2004. Deeper inside pagerank. _Internet Mathematics_ 1(3):335-380.
* Liu et al. (2010) Zhiyuan Liu, Wenyi Huang, Yabin Zheng, and Maosong Sun. 2010. Automatic keyphrase extraction via topic decomposition. In _Proceedings of the 2010 conference on empirical methods in natural language processing_. Association for Computational Linguistics, pages 366-376.
* Liu et al. (2009) Zhiyuan Liu, Peng Li, Yabin Zheng, and Maosong Sun. 2009. Clustering to find exemplar terms for keyphrase extraction. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1_. Association for Computational Linguistics, pages 257-266.
* Lopez and Romary (2010) Patrice Lopez and Laurent Romary. 2010. Humb: Automatic key term extraction from scientific articles in grobid. In _Proceedings of the 5th international workshop on semantic evaluation_. Association for Computational Linguistics, pages 248-251.
* Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into texts. Association for Computational Linguistics.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_. pages 3111-3119.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_. volume 14, pages 1532-1543.
* Wang et al. (2014) Rui Wang, Wei Liu, and Chris McDonald. 2014. Corpus-independent generic keyphrase extraction using word embedding vectors. In _Software Engineering Research Conference_. volume 39.
* Wang et al. (2015) Rui Wang, Wei Liu, and Chris McDonald. 2015. Using word embeddings to enhance keyword identification for scientific publications. In _Australasian Database Conference_. Springer, pages 257-268.
* Yin and Schutze (2014) Wennpeng Yin and Hinrich Schutze. 2014. An exploration of embeddings for generalized phrases. In _ACL (Student Research Workshop)_. pages 41-47.
* Yu and Dredze (2015) Mo Yu and Mark Dredze. 2015. Learning composition models for phrase embeddings. _Transactions of the Association for Computational Linguistics_ 3:227-242.