The rapid growth of social media platforms such as Twitter provides rich multimedia data in large scales for various research opportunities, such as sentiment analysis which focuses on automatically sentiment (positive and negative) prediction on given contents. Sentiment analysis has been widely used in real world applications by analyzing the online user-generated data, such as election prediction, opinion mining and business-related activity analysis. Emojis, which consist of various symbols ranging from cartoon facial expressions to figures such as flags and sports, are widely used in daily communications to express people's feelings . Since their first release in 2010, emojis have taken the place of emoticons (such as “:- INLINEFORM0 ” and “:-P”) BIBREF0 to create a new form of language for social media users BIBREF1 . According to recent science reports, there are 2,823 emojis in unicode standard in Emoji 11.0 , with over 50% of the Instagram posts containing one or more emojis BIBREF2 and 92% of the online population using emojis BIBREF3 .The extensive use of emojis has drawn a growing attention from researchers BIBREF4 , BIBREF5 because the emojis convey fruitful semantical and sentimental information to visually complement the textual information which is significantly useful in understanding the embedded emotional signals in texts BIBREF6 . For example, emoji embeddings have been proposed to understand the semantics behind the emojis BIBREF7 , BIBREF8 , and the embedding vectors can be used to visualize and predict emoji usages given their corresponding contexts. Previous work also shows that, it is useful to pre-train a deep neural network on an emoji prediction task with pre-trained emoji embeddings to learn the emotional signals of emojis for other tasks including sentiment, emotion and sarcasm prediction BIBREF9 . However, the previous literatures lack in considerations of the linguistic complexities and diversity of emoji. Therefore, previous emoji embedding methods fail to handle the situation when the semantics or sentiments of the learned emoji embeddings contradict the information from the corresponding contexts BIBREF5 , or when the emojis convey multiple senses of semantics and sentiments such as ( and ). In practice, emojis can either summarize and emphasis the original tune of their contexts, or express more complex semantics such as irony and sarcasm by being combined with contexts of contradictory semantics or sentiments. For the examples shown in Table TABREF3 , the emoji () is of consistent sentiment with text to emphasis the sentiment, but is of the opposite sentiment (positive) to the text sentiment (negative) example 3 and 4 to deliver a sense of sarcasm. Conventional emoji analysis can only extract single embedding of each emoji, and such embeddings will confuse the following sentiment analysis model by inconsistent sentiment signals from the input texts and emojis. Moreover, we consider the emoji effect modeling different from the conventional multimodal sentiment analysis which usually includes images and texts in that, image sentiment and text sentiment are usually assumed to be consistent BIBREF10 while it carries no such assumption for texts and emojis.To tackle such limitations, we propose a novel scheme that consists of an attention-based recurrent neural network (RNN) with robust bi-sense emoji embeddings. Inspired by the word sense embedding task in natural language processing (NLP) BIBREF11 , BIBREF12 , BIBREF13 where each sense of an ambiguous word responds to one unique embedding vector, the proposed bi-sense embedding is a more robust and fine-grained representation of the complicated semantics for emojis where each emoji is embedded into two distinct vectors, namely positive-sense and negative-sense vector, respectively. For our specific task which is Twitter sentiment analysis BIBREF14 , BIBREF15 , we initialize the bi-sense embedding vectors together with word embedding vectors using word embedding algorithm fasttext BIBREF16 by extracting two distinct embeddings for each emoji according to the sentiment of its corresponding textual contexts, namely bi-sense embedding. A long short-term memory (LSTM) based recurrent neural network is then used for predicting sentiments which is integrated with the pre-trained emoji embedding features by a context-guide and self-selected attention mechanism. Because most of the previous Twitter sentiment datasets exclude emojis and there exists little resource that contains sufficient emoji-tweets with sentiment labels, we construct our own emoji-tweets dataset by automatically generating weak labels using a rule-based sentiment analysis algorithm Vader BIBREF17 for pre-traning the networks, and manually labeling a subset of tweets for fine tuning and testing purposes. The experimental results demonstrate that the bi-sense emoji embedding is capable of extracting more distinguished information from emojis and outperforms the state-of-the-art sentiment analysis models with the proposed attention-based LSTM networks. We further visualize the bi-sense emoji embedding to obtain the sentiments and semantics learned by the proposed approach.The main contributions of this paper are summarized as follows.Sentiment analysis is to extract and quantify subjective information including the status of attitudes, emotions and opinions from a variety of contents such as texts, images and audios BIBREF18 . Sentiment analysis has been drawing great attentions because of its wide applications in business and government intelligence, political science, sociology and psychology BIBREF19 , BIBREF20 , BIBREF21 , BIBREF22 . From a technical perspective, textual sentiment analysis is first explored by researchers as an NLP task. Methods range from lexical-based approaches using features including keywords BIBREF23 , BIBREF24 where each word corresponds to a sentiment vector with entries representing the possibility of the word and each sentiment and phase-level features (n-grams and unigrams) BIBREF25 , BIBREF26 , to deep neural network based embedding approaches including skip-grams, continuous bag-of-words (CBoW) and skip-thoughts BIBREF27 , BIBREF28 , BIBREF16 , BIBREF29 . It was until recent years when researchers start focusing on image and multimodal sentiments BIBREF30 , BIBREF31 and analyzing how to take advantage of the cross-modality resources BIBREF10 , BIBREF32 . For multimodal sentiment analysis, an underlying assumption is that both modalities express similar sentiment and such similarity is enforced in order to train a robust sentiment inference model BIBREF10 . However, the same assumption does not stand in modeling textual tweets and emojis because the complexities of natural language exist extensively, such as the use of irony, jokes, sarcasm, etc. BIBREF9 .ModelsWe set up the baselines and proposed models as follows:LSTM with text embedding: CNNs and LSTMs are widely used to encode textual contents for sentiment analysis in BIBREF45 , BIBREF46 and many online tutorials. Here we select the standard LSTM with pre-trained word embedding as input, and add one fully-connected layer with sigmoid activation top of the LSTM encoder (same as all other models), denoted as T-LSTM.LSTM with emoji embedding: We consider the emoji as one special word and input both pre-trained text and emoji embeddings into the same LSTM network, namely E-LSTM. Similarly, we concatenate the pre-trained bi-sense emoji embedding as one special word to feed into the LSTM network. This model is called BiE-LSTM.Attention-based LSTM with emojis:We also use the word-emoji embedding to calculate the emoji-word attention following Equation EQREF20 and EQREF21 , and the only difference is that we replace the attention-derived senti-emoji embedding with the pre-trained word-emoji embedding by fasttext, denoted as ATT-E-LSTM.LSTM with bi-sense emoji embedding (proposed): As we have introduced in Section SECREF13 , we propose two attention-based LSTM networks based on bi-sense emoji embedding, denoted as MATT-BiE-LSTM and WATT-BiE-LSTM.EvaluationWe evaluate the baseline and proposed models on sentiment analysis by F1 scores and accuracies based on the auto-annotated testing set (AA-Sentiment) and human-annotated testing set (HA-Sentiment), as shown in Table TABREF25 . We only test the models after fine-tuning with a subset of the samples with human annotations because training exclusively on the samples with auto-generated weak labels results in relatively poor performances when tested with human annotated data indicating the models after fine-tuning are more robust. The F1 scores and accuracies are overall higher with the AA-Sentiment than the results with HA-sentiment, indicating that the HA-Sentiment is a more challenging task and the sentiments involved are more difficult to identify supported by their relatively lower sentiment scores returned from Vader. We still, however, observe competitive results from HA-Sentiment showing that the models are well-trained and robust to noisy labels with the help of fine-tuning with human annotated data. The T-LSTM baseline achieves decent performance in both experiments with accuracies of 86.6% and 70.7% showing that LSTM is an effective encoder for sentiment analysis as suggested by the references. The models with proposed bi-sense emoji embedding obtain accuracies over 82.4% and we observe improvements on the performance with the attention-based LSTM from our proposed model MATT-BiE-LSTM and WATT-BiE-LSTM, which is consistent with that ATT-E-LSTM (F1@84.6%, accuracy@82.0% on HA-Sentiment) outperforms significantly T-LSTM and E-LSTM.Emoji information is useful in sentiment analysis. Most models outperforms the baseline T-LSTM in both dataset suggesting that the emoji information is useful for sentiment analysis as a complement to the textual contents, even with the naive use of emoji embeddings (E-LSTM) when tested with HA-Sentiment. We observe that E-LSTM obtains similar performance to T-LSTM with AA-Sentiment but a significant gain over the T-LSTM when tested with HA-Sentiment indicating that sentiment information is helpful and necessary when the hidden sentiment is relatively subtle and the task is more challenging.Bi-sense emoji embedding helps. All the models using bi-sense emoji embedding perform significantly better than the baseline models without emoji feature or with word-emoji embedding. BiE-LSTM outperforms T-LSTM and E-LSTM significantly with the same utilization of emoji embedding indicates that the proposed bi-sense emoji embedding is capable of extracting more informative and distinguishable vectors over the use of conventional word embedding algorithms, which is consistent based on the comparisons between the proposed models (MATT-BiE-LSTM and WATT-BiE-LSTM) with bi-sense emoji embedding and the baseline model ATT-E-LSTM with word-emoji embedding and attention.Attention mechanism aligns and performs well with bi-sense embedding. MATT-BiE-LSTM and WATT-BiE-LSTM obtain similar performances when tested on both Vader and human annotated samples, though their ways of computing the attention (weights and vectors) are different that WATT computes attention weights and the senti-emoji embeddings guided by each word, and MATT obtains the senti-emoji embedding based on the LSTM encoder on the whole contexts and computes the attention weights of the senti-emoji embedding across all words. Both models outperforms the state-of-the-art baseline models including ATT-E-LSTM. The proposed attention-based LSTM can be further extended to handle tasks involving multi-sense embedding as inputs, such as the word-sense embedding in NLP, by using context-guide attention to self-select how much to attend on each sense of the embeddings each of which correspond to a distinct sense of semantics or sentiments. In this way we are able to take advantage of the more robust and fine-grained embeddings.With the overwhelming development of Internet of Things (IOT), the growing accessibility and popularity of subjective contents have provided new opportunities and challenges for sentiment analysis BIBREF33 . For example, social medias such as Twitter and Instagram have been explored because the massive user-generated contents with rich user sentiments BIBREF25 , BIBREF34 , BIBREF35 where emojis (and emoticons) are extensively used. Non-verbal cues of sentiment, such as emoticon which is considered as the previous generation of emoji, has been studied for their sentiment effect before emojis take over BIBREF36 , BIBREF37 , BIBREF38 . For instance, BIBREF36 , BIBREF38 pre-define sentiment labels to emoticons and construct a emoticon-sentiment dictionary. BIBREF37 applies emoticons for smoothing noisy sentiment labels. Similar work from BIBREF39 first considers emoji as a component in extracting the lexical feature for further sentiment analysis. BIBREF40 constructs an emoji sentiment ranking based on the occurrences of emojis and the human-annotated sentiments of the corresponding tweets where each emoji is assigned with a sentiment score from negative to positive , similar to the SentiWordNet BIBREF41 . However, the relatively intuitive use of emojis by lexical- and dictionary-based approaches lacks insightful understanding of the complexed semantics of emojis. Therefore, inspired by the success of word semantic embedding algorithms such as BIBREF28 , BIBREF16 , BIBREF7 obtains semantic embeddings of each emoji by averaging the words from its descriptions and shows it is effective to take advantage of the emoji embedding for the task of Twitter sentiment analysis. BIBREF8 proposes a convoluntional neural network to predict the emoji occurrence and jointly learns the emoji embedding via a matching layer based on cosine similarities. Despite the growing popularity of Twitter sentiment analysis, there is a limited number of emoji datasets with sentiment labels available because previous studies usually filter out urls, emojis and sometimes emoticons. However, BIBREF9 shows that it is effective to extract sentiment information from emojis for emotion classification and sarcasm detection tasks in the absence of learning vector-based emoji representations by pre-training a deep neural network to predict the emoji occurrence.We propose two mechanisms, namely Word-guide Attention-based LSTM and Multi-level Attention-based LSTM, to take advantage of bi-sense emoji embedding for the sentiment analysis task. The frameworks of these two methods are shown in Figure FIGREF10 and Figure FIGREF19 , respectively. Our workflow includes the following steps: initialization of bi-sense emoji embedding, generating senti-emoji embedding based on self-selected attention, and sentiment classification via the proposed attention-based LSTM networks.Recent research shows great success in word embedding task such as word2vec and fasttext BIBREF27 , BIBREF16 . We use fasttext to initialize emoji embeddings by considering each emoji as a special word, together with word embeddings. The catch is, different from conventional approaches where each emoji responds to one embedding vector (as we call word-emoji embedding), we embed each emoji into two distinct vectors (bi-sense emoji embedding): we first assign two distinct tokens to each emoji, of which one is for the particular emoji used in positive sentimental contexts and the other one is for this emoji used in negative sentimental contexts (text sentiment initialized by Vader BIBREF17 , details will be discussed in Section SECREF23 ), respectively; the same fasttext training process is used to embed each token into a distinct vector, and we thus obtain the positive-sense and negative-sense embeddings for each emoji.The word2vec is based on the skip-gram model whose objective is to maximize the log likelihood calculated by summing the probabilities of current word occurrences given a set of the surrounding words. The fasttext model is different by formatting the problem as a binary classification task to predict the occurrence of each context word, with negative samples being randomly selected from the absent context words. Given an input word sequence INLINEFORM0 , and the context word set INLINEFORM1 and the set of negative word samples INLINEFORM2 of the current word INLINEFORM3 , the objective function is obtained based on binary logistic loss as in Equation EQREF12 : DISPLAYFORM0 where INLINEFORM0 denotes the logistic loss of the score function INLINEFORM1 which is computed by summing up the scalar products between the n-gram embeddings of the current word and the context word embedding which is different from word2vec where the score is the scalar product between the current word and the context word embedding. We select fasttext over word2vec mainly because its computational efficiency. In general, the two models yield competitive performances and the comparison between word embeddings is beyond our discussion. Therefore we only show the results derived by the fasttext initialization within the scope of this work.Long short-term memory (LSTM) units have been extensively used to encode textual contents. The basic encoder model consists of a text embedding layer, LSTMs layer, and fully-connected layers for further tasks such as text classifications based on the encoded feature. The operations in an LSTM unit for time step INLINEFORM0 is formulated in Equation EQREF14 : DISPLAYFORM0 where INLINEFORM0 and INLINEFORM1 represent the current and previous hidden states, INLINEFORM2 denotes the current LSTM input and here we use the embedding INLINEFORM3 of the current word INLINEFORM4 , and INLINEFORM5 and INLINEFORM6 denote the weight matrices BIBREF42 . In order to take advantage of the bi-sense emoji embedding, we modify the input layer into the LSTM units. We first obtain the senti-emoji embedding as an weighted average of the bi-sense emoji embedding based on the self-selected attention mechanism. Let INLINEFORM7 represent the INLINEFORM8 -th sense embedding of emoji INLINEFORM9 ( INLINEFORM10 in our bi-sense embedding), and INLINEFORM11 denote the attention function conditioned on the current word embedding, the attention weight INLINEFORM12 and senti-emoji embedding vector INLINEFORM13 is formulated as follows: DISPLAYFORM0 We choose a fully-connected layer with ReLU activation as the attention function, and the attention vector INLINEFORM0 is concatenated with the word embedding as the new input of the LSTM. Thus the input vector INLINEFORM1 in Equation EQREF14 becomes INLINEFORM2 . The output of the final LSTM unit is then fed into a fully-connected layer with INLINEFORM3 activation to output the tweet sentiment and binary cross-entropy loss is used as the objection function (Equation EQREF16 ) where INLINEFORM4 is the total number of samples. The motivation behind this model is that each context word guides the attention weights in order to enforce the model to self-select which embedding sense it should attend on. Therefore we denote this model as the Word-guide Attention-based LSTM with Bi-sense emoji embedding (WATT-BiE-LSTM). DISPLAYFORM0 There is another way of formulating the attention mechanism where the attention weights indicate how the image information (which is emoji in our case) is distributed through the context words as proposed in BIBREF43 , BIBREF44 . The modified senti-emoji embedding vector INLINEFORM0 is thus at the tweet-level instead of the word-level in Equation EQREF15 by replacing the INLINEFORM1 with the final state vector INLINEFORM2 outputted from the last LSTM unit, as shown in Equation EQREF18 : DISPLAYFORM0 The derived senti-emoji embedding INLINEFORM0 is then used to calculate an additional layer of attention following BIBREF43 , BIBREF44 . Given the input tweet sequence INLINEFORM1 , the attention weight INLINEFORM2 conditioned on the senti-emoji embedding is formulated as follows: DISPLAYFORM0 Therefore, we construct the new input INLINEFORM0 to each LSTM unit by concatenating the original word embedding and the attention vector in Equation EQREF21 to distribute the senti-emoji information to each step. This model is called Multi-level Attention-based LSTM with Bi-sense Emoji Embedding (MATT-BiE-LSTM). We choose the same binary cross-entropy as the loss function with the same network configuration with WATT-BiE-LSTM. DISPLAYFORM0 Data CollectionWe construct our own Twitter sentiment dataset by crawling tweets through the REST API which consists of 350,000 users and is magnitude larger comparing to previous work. We collect up to 3,200 tweets from each user and follow the standard tweet preprocessing procedures to remove the tweets without emojis and tweets containing less than ten words, and contents including the urls, mentions, and emails.Data AnnotationFor acquiring the sentiment annotations, we first use Vader which is a rule-based sentiment analysis algorithm BIBREF17 for text tweets only to generate weak sentiment labels. The algorithm outputs sentiment scores ranging from -1 (negative) to 1 (positive) with neutral in the middle. We consider the sentiment analysis as a binary classification problem (positive sentiment and negative sentiment), we filter out samples with weak prediction scores within INLINEFORM0 and keep the tweets with strong sentiment signals. Emoji occurrences are calculated separately for positive tweets and negative tweets, and threshold is set to 2,000 to further filter out emojis which are less frequently used in at least one type of sentimental text. In the end, we have constructed a dataset with 1,492,065 tweets and 55 frequently used emojis in total. For the tweets with an absolute sentiment score over 0.70, we keep the auto-generated sentiment label as ground truth because the automatic annotation is reliable with high sentiment scores. On the other hand, we select a subset of the tweets with absolute sentiment scores between INLINEFORM1 for manual labeling by randomly sampling, following the distribution of emoji occurrences where each tweet is labeled by two graduate students. Tweets are discarded if the two annotations disagree with each other or they are labeled as neutral. In the end, we have obtained 4,183 manually labeled tweets among which 60% are used for fine-tuning and 40% are used for testing purposes. The remainder of the tweets with automatic annotations are divided into three sets: 60% are used for pre-training the bi-sense and conventional emoji embedding, 10% for validation and 30% are for testing. We do not include a “neutral” class because it is difficult to obtain valid neutral samples. For auto-generated labels, the neutrals are the samples with low absolute confidence scores and their sentiments are more likely to be model failures other than “true neutrals”. Moreover, based on the human annotations, most of the tweets with emojis convey non-neutral sentiment and only few neutral samples are observed during the manual labeling which are excluded from the manually labeled subset.In order to valid our motivation that emojis are also extensively used in tweets that contain contradictory information to the emoji sentiments, we calculate the emoji usage in Table TABREF22 according to the sentiment labels where Pos-Ratio means the percentage of each emoji occurs in the positive tweets over its total number of occurrences, AA and HA indicate automatic-annotation and human-annotation, respectively. We present the top-10 most frequently used emojis in our dataset and observe a slight difference in the Pos-Ratios between AA and HA dataset because of the randomness involved in the sampling process. Results from both of the datasets show a fair amount of emoji use in both positive and negative tweets. For example, it is interesting to notice that emoji () occurs more in the positive tweets in with the automatic annotations, while emojis with strong positive sentiment have also been used in negative tweets with about 5% occurrences, such as (, , and ). Given the averaged positive ratio among all emojis in the whole dataset is about 74% and that most emojis have been extensively used in tweets containing both positive and negative sentiments, it suggests that distinguishing the emoji occurrences in both sentiments via bi-sense embedding is worth investigating. Additionally, we observe the Pos-Ratios of the AA-sentiment and HA-sentiment have little differences which are due to two main reasons: 1) Some tweets we sampled to construct the HA-sentiment are discarded because the annotators have disagreements and we only keep the samples that we are confident about; 2) Tweets with absolute sentiment scores between (0.60,0.70) are selected for manual labeling as discussed in Section SECREF23 , which are lower than the tweets used to construct the AA-sentiment (0.7 and above). The lower sentiment scores indicate that Vader is less reliable on the samples of HA-sentiment dataset and the sentiments of these tweets are more likely to be affected by emojis.In order to obtain insights about why the more fine-grained bi-sense emoji embedding helps in understanding the complexed sentiments behind tweets, we visualize the attention weights for ATT-E-LSTM and MATT-BiE-LSTM for comparison. The example tweets with corresponding attention weights calculated by word-emoji embedding and senti-emoji embedding are shown in Figure FIGREF27 , where the contexts are presented in the captions. The emojis used are , , and , respectively.In Figure FIGREF27 (a), the ATT-E-LSTM model (baseline) assigns relatively more weights on the word “no” and “pressure”, while MATT-BiE-LSTM attends mostly on the word “happy” and “lovely”. The different attention distributions suggest that the proposed senti-emoji embedding is capable of recognizing words with strong sentiments that are closely related to the true sentiment even with the presence of words with conflicting sentiments, such as “pressure” and “happy”. while ATT-E-LSTM tends to pick up all sentimental words which could raise confusions. The senti-emoji embedding is capable of extracting representations of complexed semantics and sentiments which help guide the attentions even in cases when the word sentiment and emoji sentiment are somewhat contradictory to each other. From Figure FIGREF27 (b) and (c) we can observe that the ATT-E-LSTM assigns more weights on the sentiment-irrelevant words than the MATT-BiE-LSTM such as “hoodies”, “wait” and “after”, indicating that the proposed model is more robust to irrelevant words and concentrates better on important words. Because of the senti-emoji embedding obtained through bi-sense emoji embedding and the sentence-level LSTM encoding on the text input (described in Section SECREF13 ), we are able to construct a more robust embedding based on the semantic and sentiment information from the whole context compared to the word-emoji embedding used in ATT-E-LSTM which takes only word-level information into account.To gain further insights on the bi-sense emoji embedding, we use t-SNE BIBREF47 to project high-dimensional bi-sense embedding vectors into a two-dimensional space and preserving the relative distances between the embedding vectors at the same time. In Figure FIGREF28 we visualize the bi-sense emoji embedding, positive-sense embedding, negative-sense embedding and the subtraction between positive and negative sense embeddings of each emoji, respectively. The subtraction of an emoji between its two sense embeddings indicates the semantic differences between emoji usages in positive and negative sentimental contexts, similarly to the objective of word embeddings BIBREF28 . The positive-sense of emoji ( and ), and the negative-sense of emoji (, and ) are embedded far from the two main clusters as observed in Figure FIGREF28 (a), suggesting that the semantics of these emojis are different from the other popular emojis. The positive-sense embedding and negative-sense embeddings are clustered well with no intersection with each other. Such observation supports our objective of applying bi-sense emoji embedding because there exist such significant differences in the semantics of each emoji when appears in positive and negative sentimental contexts, and it is well-motivated to consider the emoji usages individually according to the sentiment of the contexts to extract the more fine-grained bi-sense embedding. Additionally, we observe consistent patterns in the Figure FIGREF28 (b), (c) and (d) where the sentiments conveyed in the emojis become an important factor. For example, emojis with positive sentiments such as (, and ), and emojis with negative sentiment such as (, and ) are embedded into one clusters in both positive-sense and negative-sense space. The embedding subtractions of emojis in Figure FIGREF28 (d) shows the different usages of emojis across sentiments are similar between emojis and preserve the cluster patterns observed in Figure FIGREF28 (b) and (c).In this paper, we present a novel approach to the task of sentiment analysis and achieve the state-of-the-art performance. Different from the previous work, our method combines a more robust and fine-grained bi-sense emoji embedding that effectively represents complex semantic and sentiment information, with attention-based LSTM networks that selectively attend on the correlated sense of the emoji embeddings, and seamlessly fuse the obtained senti-emoji embeddings with the word embeddings for a better understanding of the rich semantics and sentiments involved. In the future, we plan to further extend our attention-based LSTM with bi-embedding work frame to tackle tasks involving multi-sense embedding such as the learning and applications of word-sense embedding.We would like to thank the support of New York State through the Goergen Institute for Data Science, and NSF Award #1704309.

Q: Do they evaluate only on English datasets?

Q: What evidence does visualizing the attention give to show that it helps to obtain a more robust understanding of semantics and sentiments?

Q: Which SOTA models are outperformed?

Q: What is the baseline for experiments?

Q: What is the motivation for training bi-sense embeddings?