Venue: NAACL 2016

# Sentiment Composition of Words with Opposing Polarities

Svetlana Kiritchenko &Saif M. Mohammad

National Research Council Canada

{svetlana.kiritchenko,saif.mohammad}@nrc-cnrc.gc.ca

###### Abstract

In this paper, we explore sentiment composition in phrases that have at least one positive and at least one negative word--phrases like _happy accident_ and _best winter break_. We compiled a dataset of such opposing polarity phrases and manually annotated them with real-valued scores of sentiment association. Using this dataset, we analyze the linguistic patterns present in opposing polarity phrases. Finally, we apply several unsupervised and supervised techniques of sentiment composition to determine their efficacy on this dataset. Our best system, which incorporates information from the phrase's constituents, their parts of speech, their sentiment association scores, and their embedding vectors, obtains an accuracy of over 80% on the opposing polarity phrases.

## 1 Introduction

The Principle of Compositionality states that the meaning of an expression is determined by the meaning of its constituents and by its grammatical structure Montague (1974). By extension, sentiment composition is the determining of sentiment of a multi-word linguistic unit, such as a phrase or a sentence, based on its constituents. In this work, we study sentiment composition in phrases that include at least one positive and at least one negative word--for example, phrases such as _happy accident_, _couldn't stop smiling_, and _lazy sundays_. We refer to them as _opposing polarity phrases_. Such phrases present a particular challenge for automatic sentiment analysis systems that often rely on bag-of-word features.

Word-sentiment associations are commonly captured in _sentiment lexicons_. However, most existing manually created sentiment lexicons include only single words. Lexicons that include sentiment associations for _multi-word phrases_ as well as their constituent words can be very useful in studying sentiment composition. We refer to them as _sentiment composition lexicons (SCLs)_.

We created a sentiment composition lexicon for opposing polarity phrases and their constituent words Kiritchenko and Mohammad (2016c).1 Both phrases and single words were manually annotated with real-valued sentiment association scores using an annotation scheme known as Best-Worst Scaling.2 We refer to the created resource as the _Sentiment Composition Lexicon for Opposing Polarity Phrases (SCL-OPP)_. The lexicon includes entries for 265 trigrams, 311 bigrams, and 602 unigrams.

Footnote 1: www.saifmohammad.com/WebPages/SCL.html#OPP

Footnote 2: Best–Worst Scaling has been shown to produce reliable real-valued sentiment association scores Kiritchenko and Mohammad (2016a).

In this paper, we use SCL-OPP to analyze regularities present in different kinds of opposing polarity phrases. We calculate the extent to which different part-of-speech combinations result in phrases of positive and negative polarity. We also show that for most phrases, knowing the parts of speech and polarities of their constituents is not enough to reliably predict the sentiment of the phrase.

We apply several unsupervised and supervised techniques of sentiment composition to determine their efficacy on predicting the sentiment of opposing polarity phrases. Our experiments indicate that

the sentiment of the last unigram or the sentiment of the most polar unigram in the phrase are not strong predictors of the overall sentiment of the phrase. Similarly, adjectives and verbs do not always dominate the sentiment in such phrases. Finally, we show that the constituent words, their parts of speech, their sentiment association scores, and their embedding vectors are all useful features--a supervised sentiment composition system that incorporates them obtains accuracies over 80% on both bigram and trigram opposing polarity phrases.

## 2 Related Work

A number of approaches have been proposed to address sentiment composition, which include manually derived syntactic rules Moilanen and Pulman (2007); Neviarouskaya et al. (2010), combination of hand-written rules and statistical learning Choi and Cardie (2008), and machine learning approaches Nakagawa et al. (2010); Yessenalina and Cardie (2011); Dong et al. (2015). Much work has been devoted to model the impact of negators and (to a lesser degree) intensifiers, words commonly referred to as contextual valence shifters, on sentiment of words they modify Polanyi and Zaenen (2004); Kennedy and Inkpen (2005); Liu and Seneff (2009); Wiegand et al. (2010); Taboada et al. (2011); Kiritchenko et al. (2016). Kiritchenko and Mohammad (2016b) created a sentiment composition lexicon for negators, modals, and adverbs (SCL-NMA) through manual annotation and analyzed the effect of these groups of modifiers on sentiment in short phrases. Recently, recursive deep model approaches have been proposed for handling sentiment of syntactic phrases through sentiment composition over parse trees Socher et al. (2013); Zhu et al. (2014); Irsoy and Cardie (2014); Tai et al. (2015). In this work, we apply several unsupervised and supervised techniques of sentiment composition for a specific type of phrases--opposing polarity phrases.

## 3 Creating a Sentiment Lexicon for Opposing Polarity Phrases

This section summarizes how we created a sentiment composition lexicon for opposing polarity phrases using the Best-Worst Scaling annotation technique. For more details we refer the reader to Kiritchenko and Mohammad (2016c). Table 1 shows a few example entries from the lexicon.

**Term selection:** We polled the Twitter API (from 2013 to 2015) to collect about 11 million tweets that contain emoticons: '\(\cdot\)' or '\(\mathbf{:}\)('. We will refer to this corpus as the _Emoticon Tweets Corpus_. From this corpus, we selected bigrams and trigrams that had at least one positive word and at least one negative word. The polarity labels (positive or negative) of the words were determined by simple look-up in existing sentiment lexicons: Hu and Liu lexicon Hu and Liu (2004), NRC Emotion lexicon Mohammad and Turney (2010); Mohammad and Turney (2013), MPQA lexicon Wilson et al. (2005), and NRC's Twitter-specific lexicon Kiritchenko et al. (2014); Mohammad et al. (2013).3 In total, 576 opposing polarity \(n\)-grams (bigrams and trigrams) were selected. We also chose for annotation all unitograms that appeared in the selected set of bigrams and trigrams. There were 602 such unigrams. Note that even though the multi-word phrases and single-word terms were drawn from a corpus of tweets, most of the terms are used in everyday English.

Footnote 3: If a word was marked with conflicting polarity in two lexicons, then that word was not considered as positive or negative.

**Best-Worst Scaling Method of Annotation:** Best-Worst Scaling (BWS), also sometimes referred to as Maximum Difference Scaling (MaxDiff), is an annotation scheme that exploits the comparative approach to annotation Louviere and Woodworth (1990); Louviere et al. (2015). Annotators are given four items (4-tuple) and asked which term is the Best (highest in terms of the property of interest) and which is the Worst (least in terms of the property of interest). Responses to the BWS questions can then be translated into real-valued scores through a simple counting procedure: For each term, its score is calculated as the percentage of times the term was

\begin{table}
\begin{tabular}{l r} \hline
**Term** & **Sentiment score** \\ \hline best winter break & 0.844 \\ breaking free & 0.172 \\ isn’t long enough & -0.188 \\ breaking & -0.500 \\ heart breaking moment & -0.797 \\ \hline \end{tabular}
\end{table}
Table 1: Example entries in SCL–OPP.

 

chosen as the Best minus the percentage of times the term was chosen as the Worst (Orme, 2009). The scores range from -1 to 1.

We employ Best-Worst Scaling for sentiment annotation by providing four (single-word or multi-word) terms at a time and asking which term is the most positive (or least negative) and which is the least positive (or most negative). Each question was answered by eight annotators through a crowdsourcing platform, CrowdFlower.4 We refer to the resulting lexicon as the _Sentiment Composition Lexicon for Opposing Polarity Phrases (SCL-OPP)_.

Footnote 4: Let _majority answer_ refer to the option most chosen for a question. 81% of the responses matched the majority answer.

Portions of the created lexicon have been used as development and evaluation sets in SemEval-2016 Task 7 'Determining Sentiment Intensity of English and Arabic Phrases' Kiritchenko et al. (2016).5 The objective of that task was to test different methods of automatically predicting sentiment association scores for multi-word phrases.

Footnote 5: [http://alt.qcri.org/semeval2016task7/](http://alt.qcri.org/semeval2016task7/)

## 4 Sentiment Composition Patterns

SCL-OPP allows us to explore sentiment composition patterns in opposing polarity phrases. We define a _Sentiment Composition Pattern (SCP)_ as a rule that includes on the left-hand side the parts of speech (POS) and the sentiment associations of the constituent unigrams (in the order they appear in the phrase), and on the right-hand side the sentiment association of the phrase. Table 2 shows examples. SCPs that have a positive phrase on the right-hand side will be called _positive SCPs_, whereas SCPs that have a negative phrase on the right-hand side will be called _negative SCPs_. Below are some questions regarding SCPs and opposing polarity phrases that we explore here:

* Which SCPs are common among opposing polarity phrases?
* With the same left-hand side of an SCP, how often is the composed phrase positive and how often is the composed phrase negative? For example, when negative adjectives combine with a positive noun, how often is the combined phrase negative?
* Are some parts of speech (of constituent words) more influential in determining the sentiment of a phrase than others?

To answer these questions, each of the entries in SCL-OPP is marked with the appropriate SCP. The part-of-speech sequence of a phrase is determined by looking up the most common part-of-speech sequence for that phrase in the Emoticon Tweets Corpus.6 Next, for every left-hand side of an SCP, we determine the ratio of 'how often occurrences of such combinations in SCL-OPP resulted in a positive phrase' to 'how often such combinations were seen in total'. We will refer to these scores as the _occurrence rates ('Occ.') of positive SCPs_. The _occurrence rates of negative SCPs_ are calculated in a similar manner.

Footnote 6: The corpus was automatically POS tagged using the CMU Tweet NLP tool Gimpel et al. (2011).

Table 2 presents all SCPs with the left-hand side combination appearing at least ten times in SCL-OPP, and whose occurrence rate is equal to or greater than 50%. For example, the second row tells us that there are 68 bigrams in SCL-OPP such that the first word is a negative adjective and the second word is a positive noun. Out of these 68 bigrams, 59% are negative, and the remaining 41% are positive, so the occurrence rate of this pattern is 0.59.

The most common SCPs in our lexicon are "\(\triangle\)adj. + \(\bigtriangledown\rightarrow\bigtriangledown\)phrase" (73) and "\(\bigtriangledown\)adj. + \(\bigtriangleup\)noun \(\rightarrow\bigtriangledown\)phrase" (68). Observe that the occurrence rates of the patterns are spread over the entire range from 52% to 91%. Only two patterns have very high occurrence rates (around 90%): "\(\triangle\)adverb + \(\bigtriangledown\)adj. \(\rightarrow\bigtriangledown\)phrase" and "\(\bigtriangleup\)adverb + \(\bigtriangledown\)verb \(\rightarrow\bigtriangledown\)phrase".

 

Thus, for most opposing polarity phrases, their sentiment cannot be accurately determined based on the POS and sentiment of the constituents alone.

Both SCPs with high occurrence rates include adverbs that serve as intensifiers--words that increase or decrease the degree of association of the following word with positive (negative) sentiment (e.g., _incredibly slow_, _dearly missed_). Only the degree of association for the next word is changed while its polarity (positive or negative) is often preserved. Some adjectives can also play the role of an intensifier when combined with another adjective (e.g., _crazy talented_) or a noun (e.g., _epic fail_). For example, the adjective _great_, often considered highly positive, becomes an intensifier when combined with some nouns (e.g., _great loss_, _great capture_). Other adjectives determine the polarity of the entire phrase (e.g., _happy tears_, _bad luck_). Therefore, the occurrence rates of patterns like "\(\bigtriangledown\)adj. + \(\bigtriangleup\)noun \(\rightarrow\bigtriangledown\)phrase" are low. Overall, even though adjectives and verbs are frequently the primary source of sentiment in the phrase, some nouns can override their sentiment as in _new crisis_ or _leave a smile_. SCL-OPP includes phrases corresponding to many different kinds of sentiment composition patterns, and therefore, it is a useful resource for studying linguistic underpinnings of sentiment composition as well as for evaluating sentiment composition algorithms for opposing polarity phrases.

## 5 Automatically Predicting Sentiment

We now investigate whether accurate models of sentiment composition for opposing polarity phrases can be learned. We conduct experiments with several baseline unsupervised classifiers as well supervised techniques using features, such as unigrams, POS, sentiment scores, and word embeddings.

The problem of sentiment composition can be formulated in two different ways: a binary classification task where the system has to predict if the phrase is positive or negative; and a regression task where the system has to predict the real-valued sentiment association score of the phrase. We evaluate binary classification with simple accuracy (_acc._) and the regression task with Pearson correlation coefficient (\(r\)). Learning and evaluation are performed separately for bigrams and trigrams.

### Baseline Classifiers

The oracle 'majority label' baseline assigns to all instances the most frequent polarity label in the dataset. The 'last unigram' baseline returns the sentiment score (or the polarity label) of the last unigram in the phrase. For the regression task, we use the real-valued sentiment score of the unigram whereas for the binary classification task we use the polarity label (positive or negative). The 'most polar unigram' baseline assigns to the phrase the sentiment score (or the polarity label) of the most polar word in that phrase, i.e., the word with the highest absolute sentiment score. The 'part-of-speech (POS) rule' baseline assigns sentiment as follows:

1. If the phrase has an adjective, return the sentiment score (polarity) of the last adjective;
2. Else, if the phrase has a verb, return the sentiment score (polarity) of the last verb;
3. Else, return the sentiment score (polarity) of the most polar word.

### Supervised Classifiers

We train a Support Vector Machines classifier with RBF kernel for the binary classification task and a Support Vector regression model with RBF kernel for the regression task using the LibSVM package Chang and Lin (2011). For both tasks, the models are trained using different combinations of the following features obtained from the target phrase: all unigrams, POS tag of each unigram, sentiment label of each unigram, sentiment score of each unigram, and the word embedding vector for each unigram. The word embeddings are obtained by running word2vec software Mikolov et al. (2013) on the Emoticon Tweets Corpus. We use the skip-gram model with the default parameters and generate 200-dimensional vectors for each unigram present in the corpus. For each task, ten-fold cross-validation is repeated ten times, and the results are averaged.

### Results

The results for all baseline and supervised methods are presented in Table 3. The 'majority label', 'last unigram', 'most polar unigram', and 'POS rule' baselines are shown in rows a to d. Observe that the sentiment association of the last unigram is not very 

predictive of the phrase's sentiment (row b).7 Both the 'most polar unigram' and the 'POS rule' classifiers perform markedly better than the majority baseline. Interestingly, the 'most polar unigram' classifier outperforms the slightly more sophisticated 'POS rule' approach on most tasks. Also, we found that within bigram phrases that contain adjectives or verbs, the adjective or verb constituents are the most polar words in only about half of the instances (and even less so in trigrams). This indicates that adjectives and verbs do not always dominate the sentiment in a phrase.

Footnote 6: Note that the results for the ‘last unigram’ baseline are still better than the results of random guessing (acc = 50, \(r\) = 0). For the majority of \(n\)-grams in SCL-OPP, the polarity of the first unigram is opposite to the polarity of the last unigram. Thus, the results for a similar ‘first unigram’ baseline (not shown here) are worse than those obtained by the ‘last unigram’ baseline.

The results obtained using supervised techniques with various feature combinations are presented in rows e to k (Table 3). Using only POS and binary sentiment labels of the constituent unigrams, the supervised learning algorithm does not perform much better than our 'POS rule' baseline (the accuracies in row e are just slightly higher than those in row d). With access to real-valued sentiment scores of unigrams much more accurate models can be learned (row f). Furthermore, the results show that the sentiment of a phrase depends on its constituent words and not only on the sentiment of the constituents (row g shows markedly better performance than row f; all the differences are statistically significant, \(p<.01\)). Concatenating word embeddings was found to be more effective than averaging. (Averaging is common when creating features for sentences). Having access to both unigrams and word embedding features produces the best results. (The differences between the scores in row i and row j are statistically significant, \(p<.01\).) Row k shows results of the model trained without the gold sentiment scores of the unigrams. Observe that for bigrams, there is a substantial drop in performance compared to row j (6.3-point drop in accuracy on the binary task, 6.7-point drop in Pearson correlation on the regression task) whereas for trigrams the performance is not affected as much (less than 1-point change on both tasks). Thus, having access to sentiment scores of constituents is particularly useful for determining sentiment of bigram phrases.

## 6 Conclusions

We created a real-valued sentiment composition lexicon for opposing polarity phrases and their constituent words, through manual annotation. We analyzed patterns of sentiment composition across phrases formed with different POS combinations. Further, we applied several unsupervised and supervised techniques of sentiment composition to determine their efficacy on opposing polarity phrases. We showed that for most phrases the sentiment of the phrase cannot be reliably predicted only from the parts of speech and sentiment association of their constituent words, and that the constituent words, their parts of speech, their sentiment scores, and their embedding vectors are all useful features in supervised sentiment prediction on this dataset.

We intend to use SCL-OPP in the following applications: (1) to automatically create a large coverage sentiment lexicon of multi-word phrases and apply it in downstream applications such as sentence-level sentiment classification, and (2) to investigate how the human brain processes sentiment composition.

\begin{table}
\begin{tabular}{l c c c c} \hline  & **Binary** & \multicolumn{2}{c}{**Regression**} \\
**Features** & **(Acc.)** & **(Pearson r)** & The majority of \(n\)-grams in SCL-OPP, the polarity of the first unigram is opposite to the polarity of the last unigram. Thus, the results for a similar ‘first unigram’ baseline (not shown here) are worse than those obtained by the ’last unigram’ baseline. \\ \hline

- \\ b. last unigram & 57.2 & 59.3 & 0.394 & 0.376 \\ c. most polar unigram & 66.9 & 69.8 & 0.416 & 0.551 \\ d. POS rule & 65.6 & 63.8 & 0.531 & 0.515 \\

- \\ f. POS + sent. score & 74.9 & 74.8 & 0.662 & 0.578 \\ g. row f + uni & 82.0 & 81.3 & 0.764 & 0.711 \\ h. row f + emb(avg) + emb(max) & 78.2 & 79.5 & 0.763 & 0.710 \\ i. row f + emb(conc) & 80.2 & 76.5 & 0.790 & 0.719 \\ j. row f + emb(conc) + uni & **82.6** & **80.9** & **0.802** & **0.753** \\ k. POS + emb(conc) + uni & 76.3 & 80.2 & 0.735 & 0.744 \\ \end{tabular}
\end{table}
Table 3: Performance of the automatic systems on SCL-OPP. Features used: unigrams (uni), part-of-speech of a unigram (POS), sentiment binary label of a unigram (sent. label), sentiment real-valued score of a unigram (sent. score), embeddings (emb). ‘emb(conc)’ is the concatenation of the embedding vectors of the constituent unigrams; ‘emb(avg)’ is the average vector of the unigram embeddings; ‘emb(max)’ is maximal vector of the unigram embeddings.
 



## References

* [Chang and Lin2011] Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A Library for Support Vector Machines. _ACM Transactions on Intelligent Systems and Technology_, 2:27:1-27:27.
* Choi and Cardie (2008) Yejin Choi and Claire Cardie. 2008. Learning with compositional semantics as structural inference for substantial sentiment analysis. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 793-801.
* Dong et al. (2015) Li Dong, Furu Wei, Shujie Liu, Ming Zhou, and Ke Xu. 2015. A statistical parsing framework for sentiment classification. _Computational Linguistics_.
* Gimpel et al. (2011) Kevin Gimpel, Nathan Schneider, Brendan O'Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Hu and Liu (2004) Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In _Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)_, pages 168-177, New York, NY, USA.
* Irsoy and Cardie (2014) Ozan Irsoy and Claire Cardie. 2014. Deep recursive neural networks for compositionality in language. In _Advances in Neural Information Processing Systems_, pages 2096-2104.
* Kennedy and Inkpen (2005) Alistair Kennedy and Diana Inkpen. 2005. Sentiment classification of movie and product reviews using contextual valence shifters. In _Proceedings of the Workshop on the Analysis of Informal and Formal Information Exchange during Negotiations (FINEXIN)_, Ottawa, Ontario, Canada.
* Kiritchenko and Mohammad (2016a) Svetlana Kiritchenko and Saif M. Mohammad. 2016a. Capturing reliable fine-grained sentiment associations by crowdsourcing and best-worst scaling. In _Proceedings of The 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_, San Diego, California.
* Kiritchenko and Mohammad (2016b) Svetlana Kiritchenko and Saif M. Mohammad. 2016b. The effect of negators, modals, and degree adverbs on sentiment composition. In _Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA)_.
* Kiritchenko and Mohammad (2016c) Svetlana Kiritchenko and Saif M. Mohammad. 2016c. Happy accident: A sentiment composition lexicon for opposing polarity phrases. In _Proceedings of 10th edition of the the Language Resources and Evaluation Conference (LREC)_, Portoroz, Slovenia.
* Kiritchenko et al. (2014) Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014. Sentiment analysis of short informal texts. _Journal of Artificial Intelligence Research_, 50:723-762.
* Kiritchenko et al. (2016) Svetlana Kiritchenko, Saif M. Mohammad, and Mohammad Salameh. 2016. SemEval-2016 Task 7: Determining sentiment intensity of English and Arabic phrases. In _Proceedings of the International Workshop on Semantic Evaluation (SemEval)_, San Diego, California, June.
* Liu and Seneff (2009) Jingjing Liu and Stephanie Seneff. 2009. Review sentiment scoring via a parse-and-paraphrase paradigm. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 161-169.
* Louviere and Woodworth (1990) Jordan J. Louviere and George G. Woodworth. 1990. Best-worst analysis. Working Paper. Department of Marketing and Economic Analysis, University of Alberta.
* Louviere et al. (2015) Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015. _Best-Worst Scaling: Theory, Methods and Applications_. Cambridge University Press.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In _Proceedings of Workshop at ICLR_.
* Mohammad and Turney (2010) Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In _Proceedings of the NAACL-HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text_, LA, California.
* Mohammad and Turney (2013) Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word-emotion association lexicon. _Computational Intelligence_, 29(3):436-465.
* Mohammad et al. (2013) Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In _Proceedings of the International Workshop on Semantic Evaluation_, Atlanta, Georgia.
* Moilanen and Pulman (2007) Karo Moilanen and Stephen Pulman. 2007. Sentiment composition. In _Proceedings of Recent Advances in Natural Language Processing (RANLP)_, volume 7, pages 378-382.
* Montague (1974) Richard Montague. 1974. _Formal Philosophy; Selected papers of Richard Montague_. Yale University Press.
* Nakagawa et al. (2010) Tetsuji Nakagawa, Kentaro Inui, and Sadao Kurohashi. 2010. Dependency tree-based sentiment classification using CRFs with hidden variables. In _Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 786-794.
* Neviarouskaya et al. (2010) Alena Neviarouskaya, Helmut Prendinger, and Mitsuru Ishizuka. 2010. Recognition of affect, judgment,

and appreciation in text. In _Proceedings of the International Conference on Computational Linguistics_, pages 806-814.
* Orme (2009) Bryan Orme. 2009. Maxdiff analysis: Simple counting, individual-level logit, and HB. Sawtooth Software, Inc.
* Polanyi and Zaenen (2004) Livia Polanyi and Annie Zaenen. 2004. Contextual valence shifters. In _Proceedings of the Exploring Attitude and Affect in Text: Theories and Applications (AAAI Spring Symposium Series)_.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, Seattle, USA.
* Taboada et al. (2011) Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. _Computational Linguistics_, 37(2):267-307.
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_, pages 1556-1566, Beijing, China.
* Wiegand et al. (2010) Michael Wiegand, Alexandra Balahur, Benjamin Roth, Dietrich Klakow, and Andres Montoyo. 2010. A survey on the role of negation in sentiment analysis. In _Proceedings of the Workshop on Negation and Speculation in Natural Language Processing (NeSp-NLP)_, pages 60-68, Stroudsburg, PA, USA.
* Wilson et al. (2005) Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In _Proceedings of the Joint Conference on HLT and EMNLP_, pages 347-354, Stroudsburg, PA, USA.
* Yessenalina and Cardie (2011) Aniru Yessenalina and Claire Cardie. 2011. Compositional matrix-space models for sentiment analysis. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 172-182.
* Zhu et al. (2014) Xiaodan Zhu, Hongyu Guo, Saif Mohammad, and Svetlana Kiritchenko. 2014. An empirical study on the effect of negation words on sentiment. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics_, pages 304-313, Baltimore, Maryland, June.