Evaluation Strategy	One-Vs-The-Rest strategy was adopted for the task of multi-class classification and I reported F-score, micro-F, macro-F and weighted-F scores using 10-fold cross-validation. The F1 score is a weighted average of the precision and recall. In the multi-class case, this is the weighted average of the F1 score of each class. There are several types of averaging performed on the data: Micro-F calculates metrics globally by counting the total true positives, false negatives and false positives. Macro-F calculates metrics for each label, and find their unweighted mean. Macro-F does not take label imbalance into account. Weighted-F calculates metrics for each label, and find their average, weighted by support (the number of true instances for each label). Weighted-F alters macro-F to account for label imbalance.

I may conclude from this experiment that word2vec technique has the potential to capture sentiment information in the citations, but hand-crafted features have better performance.

Discussion and Conclusion	In this paper, I reported the citation sentiment classification results based on word embeddings. The binary classification results in Table TABREF27 showed that word2vec is a promising tool for distinguishing positive and negative citations. From Table TABREF27 I can see that there are no big differences among the scores generated by ACL100 and Brown100, despite they have different vocabulary sizes (ACL100 has 14,325 words, Brown100 has 56,057 words). The polarity specific word embeddings did not show its strength in the task of binary classification. For the task of classifying implicit citations (Table TABREF26 ), in general, sent2vec (macro-F 0.44) was comparable with the baseline (macro-F 0.47) and it was effective for detecting objective sentences (F-score 0.84) as well as separating X sentences from the rest (F-score 0.997), but it did not work well on distinguishing positive citations from the rest. For the overall classification (Table TABREF25 ), however, this method was not as good as hand-crafted features, such as n-grams and sentence structure features.

To compare the dimensionality of word embeddings, ACL300 gave a higher micro-F score than ACL100, but there is no difference between 300 and 100 dimensional ACL-embeddings when look at the macro-F and weighted-F scores. Table TABREF26 showed the sent2vec performance on classifying implicit citations with four categories: objective, negative, positive and excluded. The method in this experiment had a poor performance on detecting positive citations, but it was comparable with both the baseline and sentence structure method  for the category of objective citations. With respect to classifying negative citations, this method was not as good as sentence structure features but it outperformed the baseline. The results of classifying category X from the rest showed that the performances of this method and the sentence structure method are fairly equal. Table TABREF27 showed the results of classifying positive and negative citations using different word embeddings. The macro-F score 0.85 and the weighted-F score 0.86 proved that word2vec is effective on classifying positive and negative citations. However, unlike the outcomes in the paper of  , where they concluded that sentiment specific word embeddings performed best, integrating polarity information did not improve the result in this experiment.

Results	The performances of citation sentiment classification on dataset-basic and dataset-implicit were shown in Table TABREF25 and Table TABREF26 respectively. The result of classifying positive and negative citations was shown in Table TABREF27 . To compare with the outcomes in the work of  , I selected two records from their results: the best one (based on features n-gram + dependencies + negation) and the baseline (based on 1-3 grams). From Table TABREF25 I can see that the features extracted by  performed far better than word embeddings, in terms of macro-F (their best macro-F is 0.90, the one in this work is 0.33). However, the higher micro-F score (The highest micro-F in this work is 0.88, theirs is 0.78) and the weighted-F scores indicated that this method may achieve better performances if the evaluations are conducted on a balanced dataset. Among the embeddings, ACL-Embeddings performed better than Brown corpus in terms of macro-F and weighted-F measurements .

Test Dataset	To evaluate the sent2vec performance on citation sentiment detection, I conducted experiments on three datasets. The first one (dataset-basic) was originally taken from ACL Anthology  . Athar and Awais  manually annotated 8,736 citations from 310 publications in the ACL Anthology. I used all of the labeled sentences (830 positive, 280 negative and 7,626 objective) for testing.  The second dataset (dataset-implicit) was used for evaluating implicit citation classification, containing 200,222 excluded (x), 282 positive (p), 419 negative (n) and 2,880 objective (o) annotated sentences. Every sentence which does not contain any direct or indirect mention of the citation is labeled as being excluded (x) . The third dataset (dataset-pn) is a subset of dataset-basic, containing 828 positive and 280 negative citations. Dataset-pn was used for the purposes of (1) evaluating binary classification (positive versus negative) performance using sent2vec; (2) Comparing the sentiment classification ability of PS-Embeddings with other embeddings.

Training Dataset	The ACL-Embeddings (300 and 100 dimensions) from ACL collection were trained . ACL Anthology Reference Corpus contains the canonical 10,921 computational linguistics papers, from which I have generated 622,144 sentences after filtering out sentences with lower quality. For training polarity specific word embeddings (PS-Embeddings, 100 dimensions), I selected 17,538 sentences (8,769 positive and 8,769 negative) from ACL collection, by comparing sentences with the polar phrases .   The pre-trained Brown-Embeddings (100 dimensions) learned from Brown corpus was also used as a comparison.

Polarity-Specific Word Representation Training	To improve sentiment citation classification results, I trained polarity specific word embeddings (PS-Embeddings), which were inspired by the Sentiment-Specific Word Embedding  . After obtaining the PS-Embeddings, I used the same scheme to average the vectors in one sentence according to the sent2vec model.

Overall Sent2vec Training	In the work, I constructed sentence embeddings based on word embeddings. I simply averaged the vectors of the words in one sentence to obtain sentence embeddings (sent2vec). The main process in this step is to learn the word embedding matrix INLINEFORM0 :  INLINEFORM0 INLINEFORM1 INLINEFORM2 INLINEFORM3 (1) where INLINEFORM0 ( INLINEFORM1 ) is the word embedding for word INLINEFORM2 , which could be learned by the classical word2vec algorithm  . The parameters that I used to train the word embeddings are the same as in the work of Sadeghian and Sharafat

Pre-processing	The SentenceModel provided by LingPipe was used to segment raw text into its constituent sentences . The data I used to train the vectors has noise. For example, there are incomplete sentences mistakenly detected (e.g. Publication Year.). To address this issue, I eliminated sentences with less than three words.

Related Work	Mikolov et al. introduced word2vec technique  that can obtain word vectors by training text corpus. The idea of word2vec (word embeddings) originated from the concept of distributed representation of words  . The common method to derive the vectors is using neural probabilistic language model  . Word embeddings proved to be effective representations in the tasks of sentiment analysis  ,  ,  and text classification  . Sadeghian and Sharafat  extended word embeddings to sentence embeddings by averaging the word vectors in a sentiment review statement. Their results showed that word embeddings outperformed the bag-of-words model in sentiment classification. In this work, I are aiming at evaluating word embeddings for sentiment analysis of citations. The research questions are:

For example,  extracted several features for citation purpose and polarity classification, such as reference count, contrary expression and dependency relations. Jochim et al. tried to improve the result by using unigram and bigram features  .  used word level features, contextual polarity features, and sentence structure based features to detect sentiment citations. Although they generated good results using the combination of features, it required a lot of engineering work and big amount of annotated data to obtain the features. Further more, capturing accurate features relies on other NLP techniques, such as part-of-speech tagging (POS) and sentence parsing. Therefore, it is necessary to explore other techniques that are free from hand-crafted features. With the development of neural networks and deep learning, it is possible to learn the representations of concepts from unlabeled text corpus automatically. These representations can be treated as concept features for classification. An important advance in this area is the development of the word2vec technique  , which has proved to be an effective approach in Twitter sentiment classification  . In this work, the word2vec technique on sentiment analysis of citations was explored. Word embeddings trained from different corpora were compared.

Introduction	The evolution of scientific ideas happens when old ideas are replaced by new ones. Researchers usually conduct scientific experiments based on the previous publications. They either take use of others work as a solution to solve their specific problem, or they improve the results documented in the previous publications by introducing new solutions. I refer to the former as positive citation and the later negative citation. Citation sentence examples with different sentiment polarity are shown in Table TABREF2 . Sentiment analysis of citations plays an important role in plotting scientific idea flow. I can see from Table TABREF2 , one of the ideas introduced in paper A0 is Hidden Markov Model (HMM) based part-of-speech (POS) tagging, which has been referenced positively in paper A1. In paper A2, however, a better approach was brought up making the idea (HMM based POS) in paper A0 negative. This citation sentiment analysis could lead to future-works in such a way that new approaches (mentioned in paper A2) are recommended to other papers which cited A0 positively . Analyzing citation sentences during literature review is time consuming. Recently, researchers developed algorithms to automatically analyze citation sentiment.

Sentiment Analysis of Citations Using Word2vec	Citation sentiment analysis is an important task in scientific paper analysis. Existing machine learning techniques for citation sentiment analysis are focusing on labor-intensive feature engineering, which requires large annotated corpus. As an automatic feature extraction tool, word2vec has been successfully applied to sentiment analysis of short texts. In this work, I conducted empirical research with the question: how well does word2vec work on the sentiment analysis of citations? The proposed method constructed sentence vectors (sent2vec) by averaging the word embeddings, which were learned from Anthology Collections (ACL-Embeddings). I also investigated polarity-specific word embeddings (PS-Embeddings) for classifying positive and negative citations. The sentence vectors formed a feature space, to which the examined citation sentence was mapped to. Those features were input into classifiers (support vector machines) for supervised classification. Using 10-cross-validation scheme, evaluation was conducted on a set of annotated citations. The results showed that word embeddings are effective on classifying positive and negative citations. However, hand-crafted features performed better for the overall classification.

Question: What metrics are considered?
Answer: F-score, micro-F, macro-F, weighted-F