GWU NLP Lab at SemEval-2019 Task 3: EmoContext: Effective Contextual Information in Models for Emotion Detection in Sentence-level in a Multigenre Corpus	In this paper we present an emotion classifier model submitted to the SemEval-2019 Task 3: EmoContext. The task objective is to classify emotion (i.e. happy, sad, angry) in a 3-turn conversational data set. We formulate the task as a classification problem and introduce a Gated Recurrent Neural Network (GRU) model with attention layer, which is bootstrapped with contextual information and trained with a multigenre corpus. We utilize different word embeddings to empirically select the most suited one to represent our features. We train the model with a multigenre emotion corpus to leverage using all available training sets to bootstrap the results. We achieved overall %56.05 f1-score and placed 144.

Model Architecture - our model has an embedding layer of 300 dimensions using fasttext embedding, and 1024 dimensions using ELMo  embedding. GRU layer has 70 hidden unites. We have 3 perceptron layers with size 300. Last layer is a softmax layer to predict emotion tags. Textual information layers (explained in section SECREF8 ) are concatenated with GRU layer as auxiliary layer. We utilize a dropout  layer after the first perceptron layer for regularization.

Related Works	In semEval 2018 task-1, Affect in Tweets  , 6 team reported results on sub-task E-c (emotion classification), mainly using neural net architectures, features and resources, and emotion lexicons. Among these works  proposed a Bi-LSTM architecture equipped with a multi-layer self attention mechanism,  their model learned the representation of each tweet using mixture of different embedding. in WASSA 2017 Shared Task on Emotion Intensity  , among the proposed approaches, we can recognize teams who used different word embeddings: GloVe or word2vec  ,  and exploit a neural net architecture such as LSTM  ,  , LSTM-CNN combinations  ,  and bi-directional versions  to predict emotion intensity. Similar approach is developed by  using sentiment and LSTM architecture. Proper word embedding for emotion task is key, choosing the most efficient distance between vectors is crucial, the following studies explore solution sparsity related properties possibly including uniqueness  ,  .

Results and Analysis	The results indicates the impact of contextual information using different embeddings, which are different in feature representation. Results of class happy without contextual features has %44.16 by GRU-att-ELMo model, and %49.38 by GRU-att-ELMo+F. We achieved the best results combining ELMo with contextual information, and achieve %85.54 f-score overall, including class others. In this task we achieved %56.04 f-score overall for emotion classes, which indicates our model needs to improve the identification of emotion. Table TABREF22 shows our model performance on each emotion tag. The results show a low performance of the model for emotion tag happy, which is due to our data being out of domain. Most of the confusion and errors are happened among the emotion categories, which suggest further investigation and improvement. We achieved %90.48, %60.10, %60.19, %49.38 f-score for class others, angry, sad, and happy respectfully.

Word Embedding	Using different word embedding or end to end models where word representation learned from local context create different results in emotion detection. We noted that pre-trained word embeddings need to be tuned with local context during our experiments or it causes the model to not converge. We experimented with different word embedding methods such as word2vec, GloVe  , fasttext  , and ELMo. Among these methods fasttext and ELMo create better results.

Experimental Setup	We split MULTI dataset into 80%,10%,10% for train, dev, and test, respectively. We use AIT and EmoContext (data for this task) split as it is given by SemEval 2018 and semEval 2019. We describe these data sets in details in the next section. All experiments are implemented using Keras and Tensorflow in the back-end.

Introduction	In recent studies, deep learning models have achieved top performances in emotion detection and classification. Access to large amount of data has contributed to these high results. Numerous efforts have been dedicated to build emotion classification models, and successful results have been reported. In this work, we combine several popular emotional data sets in different genres, plus the one given for this task to train the emotion model we developed. We introduce a multigenre training mechanism, our intuition to combine different genres are a) to augment more training data, b) to generalize detection of emotion. We utilize Portable textual information such as subjectivity, sentiment, and presence of emotion words, because emotional sentences are subjective and affectual states like sentiment are strong indicator for presence of emotion. The rest of this paper is structured as followings: section SECREF2 introduce our neural net model, in section SECREF3 we explain the experimental setup and data that is been used for training and development sets, section SECREF4 discuss the results and analyze the errors, section SECREF5 describe related works, section SECREF6 conclude our study and discuss future direction.

Data	We used three different emotion corpora in our experiments. Our corpora are as follows: a) A multigenre corpus created by  with following genres: emotional blog posts, collected by  , headlines data set from SemEval 2007-task 14  , movie review data set  originally collected from Rotten tomatoes for sentiment analysis and it is among the benchmark sets for this task. We refer to this multigenre set as (MULTI), b) SemEval-2018 Affect in Tweets data set  (AIT) with most popular emotion tags: anger, fear, joy, and sadness, c) the data set that is given for this task, which is 3-turn conversation data. From these data sets we only used the emotion tags happy, sad, and angry. We used tag no-emotion from MULTI data set as others tag. Data statistics are shown in figures FIGREF18 , FIGREF19 , FIGREF20 . Data pre-processing - we tokenize all the data. For tweets we replace all the URLs, image URLs, hashtags, @users with specific anchors. Based on the popularity of each emoticon per each emotion tag, we replace them with the corresponding emotion tag.

Textual Information	Sentiment and objective Information (SOI)- relativity of subjectivity and sentiment with emotion are well studied in the literature. To craft these features we use SentiwordNet  , we create sentiment and subjective score per word in each sentences. SentiwordNet is the result of the automatic annotation of all the synsets of WORDNET according to the notions of positivity, negativity, and neutrality. Each synset s in WORDNET is associated to three numerical scores Pos(s), Neg(s), and Obj(s) which indicate how positive, negative, and objective (i.e., neutral) the terms contained in the synset are. Different senses of the same term may thus have different opinion-related properties. These scores are presented per sentence and their lengths are equal to the length of each sentence. In case that the score is not available, we used a fixed score 0.001. Emotion Lexicon feature (emo)- presence of emotion words is the first flag for a sentence to be emotional. We use NRC Emotion Lexicon  with 8 emotion tags (e.i. joy, trust, anticipation, surprise, anger, fear, sadness, disgust).

baseline- in each sentence we tagged every emotional word using NRC emotion lexicon  , if any emotion has majority occurrence we pick that emotion tag as sentence emotion tag, when all emotion tags happen only once we randomly choose among them, when there is no emotional word we tag the sentence as others. We only use the portion of the emotion lexicon that covers the tags in the task (i.e. happy, sad, and angry).

We normalized all the repeated characters, finally caps words are replaced with lower case but marked as caps words.

We demonstrate the presence of emotion words as an 8 dimension feature, presenting all 8 emotion categories of the NRC lexicon. Each feature represent one emotion category, where 0.001 indicates of absent of the emotion and 1 indicates the presence of the emotion. The advantage of this feature is their portability in transferring emotion learning across genres.

Model Description	Gates Recurrent Neural Network (GRU)  ,  and attention layer are used in sequential NLP problems and successful results are reported in different studies. Figure FIGREF11 shows the diagram of our model.  GRU- has been widely used in the literature to model sequential problems. RNN applies the same set of weights recursively as follow: DISPLAYFORM0  GRU is very similar to LSTM with the following equations: DISPLAYFORM0 DISPLAYFORM1  GRU has two gates, a reset gate INLINEFORM0 , and an update gate INLINEFORM1 . Intuitively, the reset gate determines how to combine the new input with the previous memory, and the update gate defines how much of the previous memory to keep around. We use Keras GRNN implementation to setup our experiments. We note that GRU units are a concatenation of GRU layers in each task. Attention layer - GRUs update their hidden state h(t) as they process a sequence and the final hidden state holds the summation of all other history information. Attention layer  modifies this process such that representation of each hidden state is an output in each GRU unit to analyze whether this is an important feature for prediction.

Conclusion and Future Direction	We combined several data sets with different annotation scheme and different genres and train an emotional deep model to classify emotion. Our results indicate that semantic and syntactic contextual features are beneficial to complex and state-of-the-art deep models for emotion detection and classification. We show that our model is able to classify non-emotion (others) with high accuracy. In future we want to improve our model to be able to distinguish between emotion classes in a more sufficient way. It is possible that hierarchical bi-directional GRU model can be beneficial, since these models compute history and future sequence while training the model.

Processing ELMo and attention is computationally very expensive, among our models GRU-att-ELMo+F has the longest training time and GRU-att-fasttext has the fastest training time. Results are shown in table TABREF21 and table refemoresultss

Training the Models	We have input size of 70 for sentence length, sentiment, and objective features and emotion lexicon feature has size 8. All these features are explained in section SECREF8 and are concatenated with GRU layer as auxiliary (input) layer. Attention comes next after GRU and have size 70. We select dropout of size 0.2. We select 30 epochs in each experiment, however, training is stopped earlier if 2 consecutive larger loss values are seen on evaluation of dev set. We use Adam  optimizer with a learning rate 0.001. We use dropout with rates 0.2. The loss function is a categorical-cross-entropy function. We use a mini batch  of size 32. All hyper-parameter values are selected empirically. We run each experiment 5 times with random initialization and report the mean score over these 5 runs. In section SECREF4 we describe how we choose the hyper-parameters values.

Question: Which word embeddings do they utilize for the EmoContext task?
Answer: ELMo, fasttext