Introduction
The emergence of social media sites with limited character constraint has ushered in a new style of communication. Twitter users within 280 characters per tweet share meaningful and informative messages. These short messages have a powerful impact on how we perceive and interact with other human beings. Their compact nature allows them to be transmitted efficiently and assimilated easily. These short messages can shape people's thought and opinion. This makes them an interesting and important area of study. Tweets are not only important for an individual but also for the companies, political parties or any organization. Companies can use tweets to gauge the performance of their products and predict market trends BIBREF0. The public opinion is particularly interesting for political parties as it gives them an idea of voter's inclination and their support. Sentiment and emotion analysis can help to gauge product perception, predict stock prices and model public opinions BIBREF1.
Sentiment analysis BIBREF2 is an important area of research in natural language processing (NLP) where we automatically determine the sentiments (positive, negative, neutral). Emotion analysis focuses on the extraction of predefined emotion from documents. Discrete emotions BIBREF3, BIBREF4 are often classified into anger, anticipation, disgust, fear, joy, sadness, surprise and trust. Sentiments and emotions are subjective and hence they are understood similarly and often used interchangeably. This is also mostly because both emotions and sentiments refer to experiences that result from the combined influences of the biological, the cognitive, and the social BIBREF5. However, emotions are brief episodes and are shorter in length BIBREF6, whereas sentiments are formed and retained for a longer period. Moreover, emotions are not always target-centric whereas sentiments are directed. Another difference between emotion and sentiment is that a sentence or a document may contain multiple emotions but a single overall sentiment.
Prior studies show that sentiment and emotion are generally tackled as two separate problems. Although sentiment and emotion are not exactly the same, they are closely related. Emotions, like joy and trust, intrinsically have an association with a positive sentiment. Similarly, anger, disgust, fear and sadness have a negative tone. Moreover, sentiment analysis alone is insufficient at times in imparting complete information. A negative sentiment can arise due to anger, disgust, fear, sadness or a combination of these. Information about emotion along with sentiment helps to better understand the state of the person or object. The close association of emotion with sentiment motivates us to build a system for sentiment analysis using the information obtained from emotion analysis.
In this paper, we put forward a robust two-layered multi-task attention based neural network which performs sentiment analysis and emotion analysis simultaneously. The model uses two levels of attention - the first primary attention builds the best representation for each word using Distributional Thesaurus and the secondary attention mechanism creates the final sentence level representation. The system builds the representation hierarchically which gives it a good intuitive working insight. We perform several experiments to evaluate the usefulness of primary attention mechanism. Experimental results show that the two-layered multi-task system for sentiment analysis which uses emotion analysis as an auxiliary task improves over the existing state-of-the-art system of SemEval 2016 Task 6 BIBREF7.
The main contributions of the current work are two-fold: a) We propose a novel two-layered multi-task attention based system for joint sentiment and emotion analysis. This system has two levels of attention which builds a hierarchical representation. This provides an intuitive explanation of its working; b) We empirically show that emotion analysis is relevant and useful in sentiment analysis. The multi-task system utilizing fine-grained information of emotion analysis performs better than the single task system of sentiment analysis.
Related Work
A survey of related literature reveals the use of both classical and deep-learning approaches for sentiment and emotion analysis. The system proposed in BIBREF8 relied on supervised statistical text classification which leveraged a variety of surface form, semantic, and sentiment features for short informal texts. A Support Vector Machine (SVM) based system for sentiment analysis was used in BIBREF9, whereas an ensemble of four different sub-systems for sentiment analysis was proposed in BIBREF10. It comprised of Long Short-Term Memory (LSTM) BIBREF11, Gated Recurrent Unit (GRU) BIBREF12, Convolutional Neural Network (CNN) BIBREF13 and Support Vector Regression (SVR) BIBREF14. BIBREF15 reported the results for emotion analysis using SVR, LSTM, CNN and Bi-directional LSTM (Bi-LSTM) BIBREF16. BIBREF17 proposed a lexicon based feature extraction for emotion text classification. A rule-based approach was adopted by BIBREF18 to extract emotion-specific semantics. BIBREF19 used a high-order Hidden Markov Model (HMM) for emotion detection. BIBREF20 explored deep learning techniques for end-to-end trainable emotion recognition. BIBREF21 proposed a multi-task learning model for fine-grained sentiment analysis. They used ternary sentiment classification (negative, neutral, positive) as an auxiliary task for fine-grained sentiment analysis (very-negative, negative, neutral, positive, very-positive). A CNN based system was proposed by BIBREF22 for three phase joint multi-task training. BIBREF23 presented a multi-task learning based model for joint sentiment analysis and semantic embedding learning tasks. BIBREF24 proposed a multi-task setting for emotion analysis based on a vector-valued Gaussian Process (GP) approach known as coregionalisation BIBREF25. A hierarchical document classification system based on sentence and document representation was proposed by BIBREF26. An attention framework for sentiment regression is described in BIBREF27. BIBREF28 proposed a DeepEmoji system based on transfer learning for sentiment, emotion and sarcasm detection through emoji prediction. However, the DeepEmoji system treats these independently, one at a time.
Our proposed system differs from the above works in the sense that none of these works addresses the problem of sentiment and emotion analysis concurrently. Our empirical analysis shows that performance of sentiment analysis is boosted significantly when this is jointly performed with emotion analysis. This may be because of the fine-grained characteristics of emotion analysis that provides useful evidences for sentiment analysis.
Proposed Methodology
We propose a novel two-layered multi-task attention based neural network for sentiment analysis where emotion analysis is utilized to improve its efficiency. Figure FIGREF1 illustrates the overall architecture of the proposed multi-task system. The proposed system consists of a Bi-directional Long Short-Term Memory (BiLSTM) BIBREF16, a two-level attention mechanism BIBREF29, BIBREF30 and a shared representation for emotion and sentiment analysis tasks. The BiLSTM encodes the word representation of each word. This representation is shared between the subsystems of sentiment and emotion analysis. Each of the shared representations is then fed to the primary attention mechanism of both the subsystems. The primary attention mechanism finds the best representation for each word for each task. The secondary attention mechanism acts on top of the primary attention to extract the best sentence representation by focusing on the suitable context for each task. Finally, the representations of both the tasks are fed to two different feed-forward neural networks to produce two outputs - one for sentiment analysis and one for emotion analysis. Each component is explained in the subsequent subsections.
Proposed Methodology ::: Two-Layered Multi-Task Attention Model ::: BiLSTM based word encoder
Recurrent Neural Networks (RNN) are a class of networks which take sequential input and computes a hidden state vector for each time step. The current hidden state vector depends on the current input and the previous hidden state vector. This makes them good for handling sequential data. However, they suffer from a vanishing or exploding gradient problem when presented with long sequences. The gradient for back-propagating error either reduces to a very small number or increases to a very high value which hinders the learning process. Long Short Term Memory (LSTM) BIBREF11, a variant of RNN solves this problem by the gating mechanisms. The input, forget and output gates control the information flow.
BiLSTM is a special type of LSTM which takes into account the output of two LSTMs - one working in the forward direction and one working in the backward direction. The presence of contextual information for both past and future helps the BiLSTM to make an informed decision. The concatenation of a hidden state vectors $\overrightarrow{h_t}$ of the forward LSTM and $\overleftarrow{h_t}$ of the backward LSTM at any time step t provides the complete information. Therefore, the output of the BiLSTM at any time step t is $h_t$ = [$\overrightarrow{h_t}$, $\overleftarrow{h_t}$]. The output of the BiLSTM is shared between the main task (Sentiment Analysis) and the auxiliary task (Emotion Analysis).
Proposed Methodology ::: Two-Layered Multi-Task Attention Model ::: Word Attention
The word level attention (primary attention) mechanism gives the model a flexibility to represent each word for each task differently. This improves the word representation as the model chooses the best representation for each word for each task. A Distributional Thesaurus (DT) identifies words that are semantically similar, based on whether they tend to occur in a similar context. It provides a word expansion list for words based on their contextual similarity. We use the top-4 words for each word as their candidate terms. We only use the top-4 words for each word as we observed that the expansion list with more words started to contain the antonyms of the current word which empirically reduced the system performance. Word embeddings of these four candidate terms and the hidden state vector $h_t$ of the input word are fed to the primary attention mechanism. The primary attention mechanism finds the best attention coefficient for each candidate term. At each time step $t$ we get V($x_t$) candidate terms for each input $x_t$ with $v_i$ being the embedding for each term (Distributional Thesaurus and word embeddings are described in the next section). The primary attention mechanism assigns an attention coefficient to each of the candidate terms having the index $i$ $\in $ V($x_t$):
where $W_w$ and $b_{w}$ are jointly learned parameters.
Each embedding of the candidate term is weighted with the attention score $\alpha _{ti}$ and then summed up. This produces $m_{t}$, the representation for the current input $x_{t}$ obtained from the Distributional Thesaurus using the candidate terms.
Finally, $m_{t}$ and $h_{t}$ are concatenated to get $\widehat{h_{t}}$, the final output of the primary attention mechanism.
Proposed Methodology ::: Two-Layered Multi-Task Attention Model ::: Sentence Attention
The sentence attention (secondary attention) part focuses on each word of the sentence and assigns the attention coefficients. The attention coefficients are assigned on the basis of words' importance and their contextual relevance. This helps the model to build the overall sentence representation by capturing the context while weighing different word representations individually. The final sentence representation is obtained by multiplying each word vector representation with their attention coefficient and summing them over. The attention coefficient $\alpha _t$ for each word vector representation and the sentence representation $\widehat{H}$ are calculated as:
where $W_s$ and $b_{s}$ are parameters to be learned.
$\widehat{H}$ denotes the sentence representation for sentiment analysis. Similarly, we calculate $\bar{H}$ which represents the sentence for emotion classification. The system has the flexibility to compute different representations for sentiment and emotion analysis both.
Proposed Methodology ::: Two-Layered Multi-Task Attention Model ::: Final Output
The final outputs for both sentiment and emotion analysis are computed by feeding $\widehat{H}$ and $\bar{H}$ to two different one-layer feed forward neural networks. For our task, the feed forward network for sentiment analysis has two output units, whereas the feed forward network for emotion analysis has eight output nodes performing multi-label classification.
Proposed Methodology ::: Distributional Thesaurus
Distributional Thesaurus (DT) BIBREF31 ranks words according to their semantic similarity. It is a resource which produces a list of words in decreasing order of their similarity for each word. We use the DT to expand each word of the sentence. The top-4 words serve as the candidate terms for each word. For example, the candidate terms for the word good are: great, nice awesome and superb. DT offers the primary attention mechanism external knowledge in the form of candidate terms. It assists the system to perform better when presented with unseen words during testing as the unseen words could have been a part of the DT expansion list. For example, the system may not come across the word superb during training but it can appear in the test set. Since the system has already seen the word superb in the DT expansion list of the word good, it can handle this case efficiently. This fact is established by our evaluation results as the model performs better when the DT expansion and primary attentions are a part of the final multi-task system.
Proposed Methodology ::: Word Embeddings
Word embeddings represent words in a low-dimensional numerical form. They are useful for solving many NLP problems. We use the pre-trained 300 dimensional Google Word2Vec BIBREF32 embeddings. The word embedding for each word in the sentence is fed to the BiLSTM network to get the current hidden state. Moreover, the primary attention mechanism is also applied to the word embeddings of the candidate terms for the current word.
Datasets, Experiments and Analysis
In this section we present the details of the datasets used for the experiments, results that we obtain and the necessary analysis.
Datasets, Experiments and Analysis ::: Datasets
We evaluate our proposed approach for joint sentiment and emotion analysis on the benchmark dataset of SemEval 2016 Task 6 BIBREF7 and Stance Sentiment Emotion Corpus (SSEC) BIBREF15. The SSEC corpus is an annotation of the SemEval 2016 Task 6 corpus with emotion labels. The re-annotation of the SemEval 2016 Task 6 corpus helps to bridge the gap between the unavailability of a corpus with sentiment and emotion labels. The SemEval 2016 corpus contains tweets which are classified into positive, negative or other. It contains 2,914 training and 1,956 test instances. The SSEC corpus is annotated with anger, anticipation, disgust, fear, joy, sadness, surprise and trust labels. Each tweet could belong to one or more emotion classes and one sentiment class. Table TABREF15 shows the data statistics of SemEval 2016 task 6 and SSEC which are used for sentiment and emotion analysis, respectively.
Datasets, Experiments and Analysis ::: Preprocessing
The SemEval 2016 task 6 corpus contains tweets from Twitter. Since the tweets are derived from an environment with the constraint on the number of characters, there is an inherent problem of word concatenation, contractions and use of hashtags. Example: #BeautifulDay, we've, etc. Usernames and URLs do not impart any sentiment and emotion information (e.g. @John). We use the Python package ekphrasis BIBREF33 for handling these situations. Ekphrasis helps to split the concatenated words into individual words and expand the contractions. For example, #BeautifulDay to # Beautiful Day and we've to we have. We replace usernames with $<$user$>$, number with $<number>$ and URLs with $<$url$>$ token.
Datasets, Experiments and Analysis ::: Implementation Details
We implement our model in Python using Tensorflow on a single GPU. We experiment with six different BiLSTM based architectures. The three architectures correspond to BiLSTM based systems without primary attention i.e. only with secondary attention for sentiment analysis (S1), emotion analysis (E1) and the multi-task system (M1) for joint sentiment and emotion analysis. The remaining three architectures correspond to the systems for sentiment analysis (S2), emotion analysis (E2) and multi-task system (M2), with both primary and secondary attention. The weight matrices were initialized randomly using numbers form a truncated normal distribution. The batch size was 64 and the dropout BIBREF34 was 0.6 with the Adam optimizer BIBREF35. The hidden state vectors of both the forward and backward LSTM were 300-dimensional, whereas the context vector was 150-dimensional. Relu BIBREF36 was used as the activation for the hidden layers, whereas in the output layer we used sigmoid as the activation function. Sigmoid cross-entropy was used as the loss function. F1-score was reported for the sentiment analysis BIBREF7 and precision, recall and F1-score were used as the evaluation metric for emotion analysis BIBREF15. Therefore, we report the F1-score for sentiment and precision, recall and F1-score for emotion analysis.
Datasets, Experiments and Analysis ::: Results and Analysis
We compare the performance of our proposed system with the state-of-the-art systems of SemEval 2016 Task 6 and the systems of BIBREF15. Experimental results show that the proposed system improves the existing state-of-the-art systems for sentiment and emotion analysis. We summarize the results of evaluation in Table TABREF18.
The primary attention mechanism plays a key role in the overall system as it improves the score of both sentiment and emotion analysis in both single task as well as multi-task systems. The use of primary attention improves the performance of single task systems for sentiment and emotion analysis by 2.21 and 1.72 points, respectively.Similarly, when sentiment and emotion analysis are jointly performed the primary attention mechanism improves the score by 0.93 and 2.42 points for sentiment and emotion task, respectively. To further measure the usefulness of the primary attention mechanism and the Distributional Thesaurus, we remove it from the systems S2, E2, and M2 to get the systems S1, E1, and M1. In all the cases, with the removal of primary attention mechanism, the performance drops. This is clearly illustrated in Figure FIGREF21. These observations indicate that the primary attention mechanism is an important component of the two-layered multi-task attention based network for sentiment analysis. We also perform t-test BIBREF40 for computing statistical significance of the obtained results from the final two-layered multi-task system M2 for sentiment analysis by calculating the p-values and observe that the performance gain over M1 is significant with p-value = 0.001495. Similarly, we perform the statistical significance test for each emotion class. The p-values for anger, anticipation, fear, disgust, joy, sadness, surprise and trust are 0.000002, 0.000143, 0.00403, 0.000015, 0.004607, 0.069, 0.000001 and 0.000001, respectively. These results provide a good indication of statistical significance.
Table TABREF19 shows the comparison of our proposed system with the existing state-of-the-art system of SemEval 2016 Task 6 for the sentiment dataset. BIBREF7 used feature-based SVM, BIBREF39 used keyword rules, LitisMind relied on hashtag rules on external data, BIBREF38 utilized a combination of sentiment classifiers and rules, whereas BIBREF37 used a maximum entropy classifier with domain-specific features. Our system comfortably surpasses the existing best system at SemEval. Our system manages to improve the existing best system of SemEval 2016 task 6 by 3.2 F-score points for sentiment analysis.
We also compare our system with the state-of-the-art systems proposed by BIBREF15 on the emotion dataset. The comparison is demonstrated in Table TABREF22. Maximum entropy, SVM, LSTM, Bi-LSTM, and CNN were the five individual systems used by BIBREF15. Overall, our proposed system achieves an improvement of 5 F-Score points over the existing state-of-the-art system for emotion analysis. Individually, the proposed system improves the existing F-scores for all the emotions except surprise. The findings of BIBREF15 also support this behavior (i.e. worst result for the surprise class). This could be attributed to the data scarcity and a very low agreement between the annotators for the emotion surprise.
Experimental results indicate that the multi-task system which uses fine-grained information of emotion analysis helps to boost the performance of sentiment analysis. The system M1 comprises of the system S1 performing the main task (sentiment analysis) with E1 undertaking the auxiliary task (emotion analysis). Similarly, the system M2 is made up of S2 and E2 where S2 performs the main task (sentiment analysis) and E2 commits to the auxiliary task (emotion analysis). We observe that in both the situations, the auxiliary task, i.e. emotional information increases the performance of the main task, i.e. sentiment analysis when these two are jointly performed. Experimental results help us to establish the fact that emotion analysis benefits sentiment analysis. The implicit sentiment attached to the emotion words assists the multi-task system. Emotion such as joy and trust are inherently associated with a positive sentiment whereas, anger, disgust, fear and sadness bear a negative sentiment. Figure FIGREF21 illustrates the performance of various models for sentiment analysis.
As a concrete example which justifies the utility of emotion analysis in sentiment analysis is shown below.
@realMessi he is a real sportsman and deserves to be the skipper.
The gold labels for the example are anticipation, joy and trust emotion with a positive sentiment. Our system S2 (single task system for sentiment analysis with primary and secondary attention) had incorrectly labeled this example with a negative sentiment and the E2 system (single task system with both primary and secondary attention for emotion analysis) had tagged it with anticipation and joy only. However, M2 i.e. the multi-task system for joint sentiment and emotion analysis had correctly classified the sentiment as positive and assigned all the correct emotion tags. It predicted the trust emotion tag, in addition to anticipation and joy (which were predicted earlier by E2). This helped M2 to correctly identify the positive sentiment of the example. The presence of emotional information helped the system to alter its sentiment decision (negative by S2) as it had better understanding of the text.
A sentiment directly does not invoke a particular emotion always and a sentiment can be associated with more than one emotion. However, emotions like joy and trust are associated with positive sentiment mostly whereas, anger, disgust and sadness are associated with negative sentiment particularly. This might be the reason of the extra sentiment information not helping the multi-task system for emotion analysis and hence, a decreased performance for emotion analysis in the multi-task setting.
Datasets, Experiments and Analysis ::: Error Analysis
We perform quantitative error analysis for both sentiment and emotion for the M2 model. Table TABREF23 shows the confusion matrix for sentiment analysis. anger,anticipation,fear,disgust,joy,sadness,surprise,trust consist of the confusion matrices for anger, anticipation, fear, disgust, joy, sadness, surprise and trust. We observe from Table TABREF23 that the system fails to label many instances with the emotion surprise. This may be due to the reason that this particular class is the most underrepresented in the training set. A similar trend can also be observed for the emotion fear and trust in Table TABREF23 and Table TABREF23, respectively. These three emotions have the least share of training instances, making the system less confident towards these emotions.
Moreover, we closely analyze the outputs to understand the kind of errors that our proposed model faces. We observe that the system faces difficulties at times and wrongly predicts the sentiment class in the following scenarios:
$\bullet $ Often real-world phrases/sentences have emotions of conflicting nature. These conflicting nature of emotions are directly not evident from the surface form and are left unsaid as these are implicitly understood by humans. The system gets confused when presented with such instances.
Text: When you become a father you realize that you are not the most important person in the room anymore... Your child is!
Actual Sentiment: positive
Actual Emotion: anticipation, joy, surprise, trust
Predicted Sentiment: negative
Predicted Emotion: anger, anticipation, sadness
The realization of not being the most important person in a room invokes anger, anticipation and sadness emotions, and a negative sentiment. However, it is a natural feeling of overwhelmingly positive sentiment when you understand that your own child is the most significant part of your life.
$\bullet $ Occasionally, the system focuses on the less significant part of the sentences. Due to this the system might miss crucial information which can influence and even change the final sentiment or emotion. This sometimes lead to the incorrect prediction of the overall sentiment and emotion.
Text: I've been called many things, quitter is not one of them...
Actual Sentiment: positive
Actual Emotion: anticipation, joy, trust
Predicted Sentiment: negative
Predicted Emotion: anticipation, sadness
Here, the system focuses on the first part of the sentence where the speaker was called many things which denotes a negative sentiment. Hence, the system predicts a negative sentiment and, anticipation and sadness emotions. However, the speaker in the second part uplifts the overall tone by justifying that s/he has never been called a quitter. This changes the negative sentiment to a positive sentiment and the overall emotion.
Conclusion
In this paper, we have presented a novel two-layered multi-task attention based neural network which performs sentiment analysis through emotion analysis. The primary attention mechanism of the two-layered multi-task system relies on Distributional Thesaurus which acts as a source of external knowledge. The system hierarchically builds the final representation from the word level to the sentence level. This provides a working insight to the system and its ability to handle the unseen words. Evaluation on the benchmark dataset suggests an improvement of 3.2 F-score point for sentiment analysis and an overall performance boost of 5 F-score points for emotion analysis over the existing state-of-the-art systems. The system empirically establishes the fact that emotion analysis is both useful and relevant to sentiment analysis. The proposed system does not rely on any language dependent features or lexicons. This makes it extensible to other languages as well. In future, we would like to extend the two-layered multi-task attention based neural network to other languages.
Acknowledgements
Asif Ekbal acknowledges the Young Faculty Research Fellowship (YFRF), supported by Visvesvaraya PhD scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia).

Question: What was their result on Stance Sentiment Emotion Corpus?
Answer: F1 score of 66.66%

Question: What performance did they obtain on the SemEval dataset?
Answer: F1 score of 82.10%

Question: What are the state-of-the-art systems?
Answer: For sentiment analysis UWB, INF-UFRGS-OPINION-MINING, LitisMind, pkudblab and SVM + n-grams + sentiment and for emotion analysis MaxEnt, SVM, LSTM, BiLSTM and CNN

Question: How is multi-tasking performed?
Answer: The proposed system consists of a Bi-directional Long Short-Term Memory (BiLSTM) BIBREF16, a two-level attention mechanism BIBREF29, BIBREF30 and a shared representation for emotion and sentiment analysis tasks., Each of the shared representations is then fed to the primary attention mechanism

Question: What are the datasets used for training?
Answer: SemEval 2016 Task 6 BIBREF7, Stance Sentiment Emotion Corpus (SSEC) BIBREF15

Question: How many parameters does the model have?
Answer: Unanswerable

Question: What is the previous state-of-the-art model?
Answer: BIBREF7, BIBREF39, BIBREF37, LitisMind, Maximum entropy, SVM, LSTM, Bi-LSTM, and CNN

Question: What is the previous state-of-the-art performance?
Answer: Unanswerable