Abstract: Sentiment analysis aims to uncover emotions conveyed through information. In its simplest form, it is performed on a polarity basis, where the goal is to classify information with positive or negative emotion. Recent research has explored more nuanced ways to capture emotions that go beyond polarity. For these methods to work, they require a critical resource: a lexicon that is appropriate for the task at hand, in terms of the range of emotions it captures diversity. In the past, sentiment analysis lexicons have been created by experts, such as linguists and behavioural scientists, with strict rules. Lexicon evaluation was also performed by experts or gold standards. In our paper, we propose a crowdsourcing method for lexicon acquisition, which is scalable, cost-effective, and doesn't require experts or gold standards. We also compare crowd and expert evaluations of the lexicon, to assess the overall lexicon quality, and the evaluation capabilities of the crowd.

Introduction: Sentiment analysis aims to uncover the emotion conveyed through information. In online social networks, sentiment analysis is mainly performed for political and marketing purposes, product acceptance and feedback systems. This involves the analysis of various social media information types, such as text BIBREF0 , emoticons and hashtags, or multimedia BIBREF1 . However, to perform sentiment analysis, information has to be labelled with a sentiment. This relationship is defined in a lexicon. Lexicon acquisition is a requirement for sentiment classification. During the acquisition process, individual or grouped information elements are labelled based on a class, usually an emotion. Sentiment classification is the task that uses the acquired lexicon and a classification method to classify a sentence, phrase, or social media submission as a whole, based on the aggregation of its labels. Thus, lexicon quality directly affects sentiment classification accuracy. Both tasks can either be performed automatically BIBREF2 or manually BIBREF3 where the labelling by linguists or researchers themselves BIBREF4 . Apart from experts, manual labbeling can also be performed with the help of a wide network of people, known as crowdsourcing BIBREF5 . Crowdsourcing is widely used for polarity lexicons, but rarely for beyond polarity and never for the discovery of linguistic elements. Sentiment analysis is commonly performed in polarity basis, i.e. the distinction between positive and negative emotion . These poles correspond to agreement and disagreement, or acceptance and disapproval, for candidates and products repsectively BIBREF6 . Beyond polarity (also known as pure emotion) sentiment analysis aims to uncover an exact emotion, based on emotional theories BIBREF7 , BIBREF8 . Applications such as sentiment tracking, marketing, text correction, and text to speech systems can be improved with the use of distinct emotion lexicons. However, beyond polarity studies acquire lexicons based on a set of strict rules, and the evaluation of experts. These lexicons use only a single emotion per term BIBREF9 . The problems of these approaches is the lack of uniformity and contribution freedom when relying on gold standards, and high costs with low scalability when employing experts. Natural Language Processing (NLP) applications that only rely on experts are less comprehensive, restricted, and not scalable, compared to crowdsourced NLP applications BIBREF10 . This paper presents our approach for the acquisition of a multiclass and scalable crowdsourced pure emotion lexicon (PEL), based on Plutchik's eight basic emotions. Furthermore, the crowd is also responsible for identifying linguistic elements, namely intensifiers, negators, and stop words. Usually these elements are pooled from existing lists BIBREF11 created by experts. We also introduce a worker filtering method to identify and exclude dishonest or spamming contributors, that doesn't require gold standards. Our goal is to maintain an end to end automated work-flow for a crowdsourced (annotation and evaluation wise) lexicon acquisition process. Therefore, to highlight crowd's performance on evaluation, we compare evaluations from linguistic experts and the crowd itself.

Question: How do they compare lexicons?
Answer: Human evaluators were asked to evaluate on a scale from 1 to 5 the validity of the lexicon annotations made by the experts and crowd contributors.