Introduction
In recent studies, deep learning models have achieved top performances in emotion detection and classification. Access to large amount of data has contributed to these high results. Numerous efforts have been dedicated to build emotion classification models, and successful results have been reported. In this work, we combine several popular emotional data sets in different genres, plus the one given for this task to train the emotion model we developed. We introduce a multigenre training mechanism, our intuition to combine different genres are a) to augment more training data, b) to generalize detection of emotion. We utilize Portable textual information such as subjectivity, sentiment, and presence of emotion words, because emotional sentences are subjective and affectual states like sentiment are strong indicator for presence of emotion.
The rest of this paper is structured as followings: section SECREF2 introduce our neural net model, in section SECREF3 we explain the experimental setup and data that is been used for training and development sets, section SECREF4 discuss the results and analyze the errors, section SECREF5 describe related works, section SECREF6 conclude our study and discuss future direction.
Model Description
Gates Recurrent Neural Network (GRU) BIBREF0 , BIBREF1 and attention layer are used in sequential NLP problems and successful results are reported in different studies. Figure FIGREF11 shows the diagram of our model.
GRU- has been widely used in the literature to model sequential problems. RNN applies the same set of weights recursively as follow: DISPLAYFORM0
GRU is very similar to LSTM with the following equations: DISPLAYFORM0 DISPLAYFORM1
GRU has two gates, a reset gate INLINEFORM0 , and an update gate INLINEFORM1 . Intuitively, the reset gate determines how to combine the new input with the previous memory, and the update gate defines how much of the previous memory to keep around. We use Keras GRNN implementation to setup our experiments. We note that GRU units are a concatenation of GRU layers in each task.
Attention layer - GRUs update their hidden state h(t) as they process a sequence and the final hidden state holds the summation of all other history information. Attention layer BIBREF2 modifies this process such that representation of each hidden state is an output in each GRU unit to analyze whether this is an important feature for prediction.
Model Architecture - our model has an embedding layer of 300 dimensions using fasttext embedding, and 1024 dimensions using ELMo BIBREF3 embedding. GRU layer has 70 hidden unites. We have 3 perceptron layers with size 300. Last layer is a softmax layer to predict emotion tags. Textual information layers (explained in section SECREF8 ) are concatenated with GRU layer as auxiliary layer. We utilize a dropout BIBREF4 layer after the first perceptron layer for regularization.
Textual Information
Sentiment and objective Information (SOI)- relativity of subjectivity and sentiment with emotion are well studied in the literature. To craft these features we use SentiwordNet BIBREF5 , we create sentiment and subjective score per word in each sentences. SentiwordNet is the result of the automatic annotation of all the synsets of WORDNET according to the notions of positivity, negativity, and neutrality. Each synset s in WORDNET is associated to three numerical scores Pos(s), Neg(s), and Obj(s) which indicate how positive, negative, and objective (i.e., neutral) the terms contained in the synset are. Different senses of the same term may thus have different opinion-related properties. These scores are presented per sentence and their lengths are equal to the length of each sentence. In case that the score is not available, we used a fixed score 0.001.
Emotion Lexicon feature (emo)- presence of emotion words is the first flag for a sentence to be emotional. We use NRC Emotion Lexicon BIBREF6 with 8 emotion tags (e.i. joy, trust, anticipation, surprise, anger, fear, sadness, disgust). We demonstrate the presence of emotion words as an 8 dimension feature, presenting all 8 emotion categories of the NRC lexicon. Each feature represent one emotion category, where 0.001 indicates of absent of the emotion and 1 indicates the presence of the emotion. The advantage of this feature is their portability in transferring emotion learning across genres.
Word Embedding
Using different word embedding or end to end models where word representation learned from local context create different results in emotion detection. We noted that pre-trained word embeddings need to be tuned with local context during our experiments or it causes the model to not converge. We experimented with different word embedding methods such as word2vec, GloVe BIBREF7 , fasttext BIBREF8 , and ELMo. Among these methods fasttext and ELMo create better results.
Experimental Setup
We split MULTI dataset into 80%,10%,10% for train, dev, and test, respectively. We use AIT and EmoContext (data for this task) split as it is given by SemEval 2018 and semEval 2019. We describe these data sets in details in the next section. All experiments are implemented using Keras and Tensorflow in the back-end.
Data
We used three different emotion corpora in our experiments. Our corpora are as follows: a) A multigenre corpus created by BIBREF9 with following genres: emotional blog posts, collected by BIBREF10 , headlines data set from SemEval 2007-task 14 BIBREF11 , movie review data set BIBREF12 originally collected from Rotten tomatoes for sentiment analysis and it is among the benchmark sets for this task. We refer to this multigenre set as (MULTI), b) SemEval-2018 Affect in Tweets data set BIBREF13 (AIT) with most popular emotion tags: anger, fear, joy, and sadness, c) the data set that is given for this task, which is 3-turn conversation data. From these data sets we only used the emotion tags happy, sad, and angry. We used tag no-emotion from MULTI data set as others tag. Data statistics are shown in figures FIGREF18 , FIGREF19 , FIGREF20 .
Data pre-processing - we tokenize all the data. For tweets we replace all the URLs, image URLs, hashtags, @users with specific anchors. Based on the popularity of each emoticon per each emotion tag, we replace them with the corresponding emotion tag. We normalized all the repeated characters, finally caps words are replaced with lower case but marked as caps words.
Training the Models
We have input size of 70 for sentence length, sentiment, and objective features and emotion lexicon feature has size 8. All these features are explained in section SECREF8 and are concatenated with GRU layer as auxiliary (input) layer. Attention comes next after GRU and have size 70. We select dropout of size 0.2. We select 30 epochs in each experiment, however, training is stopped earlier if 2 consecutive larger loss values are seen on evaluation of dev set. We use Adam BIBREF14 optimizer with a learning rate 0.001. We use dropout with rates 0.2. The loss function is a categorical-cross-entropy function. We use a mini batch BIBREF15 of size 32. All hyper-parameter values are selected empirically. We run each experiment 5 times with random initialization and report the mean score over these 5 runs. In section SECREF4 we describe how we choose the hyper-parameters values.
baseline- in each sentence we tagged every emotional word using NRC emotion lexicon BIBREF6 , if any emotion has majority occurrence we pick that emotion tag as sentence emotion tag, when all emotion tags happen only once we randomly choose among them, when there is no emotional word we tag the sentence as others. We only use the portion of the emotion lexicon that covers the tags in the task (i.e. happy, sad, and angry).
Results and Analysis
The results indicates the impact of contextual information using different embeddings, which are different in feature representation. Results of class happy without contextual features has %44.16 by GRU-att-ELMo model, and %49.38 by GRU-att-ELMo+F.
We achieved the best results combining ELMo with contextual information, and achieve %85.54 f-score overall, including class others. In this task we achieved %56.04 f-score overall for emotion classes, which indicates our model needs to improve the identification of emotion. Table TABREF22 shows our model performance on each emotion tag. The results show a low performance of the model for emotion tag happy, which is due to our data being out of domain. Most of the confusion and errors are happened among the emotion categories, which suggest further investigation and improvement. We achieved %90.48, %60.10, %60.19, %49.38 f-score for class others, angry, sad, and happy respectfully.
Processing ELMo and attention is computationally very expensive, among our models GRU-att-ELMo+F has the longest training time and GRU-att-fasttext has the fastest training time. Results are shown in table TABREF21 and table refemoresultss
Related Works
In semEval 2018 task-1, Affect in Tweets BIBREF13 , 6 team reported results on sub-task E-c (emotion classification), mainly using neural net architectures, features and resources, and emotion lexicons. Among these works BIBREF16 proposed a Bi-LSTM architecture equipped with a multi-layer self attention mechanism, BIBREF17 their model learned the representation of each tweet using mixture of different embedding. in WASSA 2017 Shared Task on Emotion Intensity BIBREF18 , among the proposed approaches, we can recognize teams who used different word embeddings: GloVe or word2vec BIBREF19 , BIBREF20 and exploit a neural net architecture such as LSTM BIBREF21 , BIBREF22 , LSTM-CNN combinations BIBREF23 , BIBREF24 and bi-directional versions BIBREF19 to predict emotion intensity. Similar approach is developed by BIBREF25 using sentiment and LSTM architecture. Proper word embedding for emotion task is key, choosing the most efficient distance between vectors is crucial, the following studies explore solution sparsity related properties possibly including uniqueness BIBREF26 , BIBREF27 .
Conclusion and Future Direction
We combined several data sets with different annotation scheme and different genres and train an emotional deep model to classify emotion. Our results indicate that semantic and syntactic contextual features are beneficial to complex and state-of-the-art deep models for emotion detection and classification. We show that our model is able to classify non-emotion (others) with high accuracy.
In future we want to improve our model to be able to distinguish between emotion classes in a more sufficient way. It is possible that hierarchical bi-directional GRU model can be beneficial, since these models compute history and future sequence while training the model.

Question: Do they treat differerent turns of conversation differently when modeling features?
Answer: No

Unanswerable

Question: How do they bootstrap with contextual information?
Answer: pre-trained word embeddings need to be tuned with local context during our experiments

Unanswerable

Question: Which word embeddings do they utilize for the EmoContext task?
Answer: ELMo, fasttext

word2vec, GloVe BIBREF7, fasttext BIBREF8, ELMo