VISU at WASSA 2023 Shared Task: Detecting Emotions in Reaction to News Stories Using Transformers and Stacked Embeddings
Year: 2023

Abstract:
Our system, VISU, participated in the WASSA 2023 Shared Task (3) of Emotion Classification from essays written in reaction to news articles. Emotion detection from complex dialogues is challenging and often requires context/domain understanding. Therefore in this research, we have focused on developing deep learning (DL) models using the combination of word embedding representations with tailored prepossessing strategies to capture the nuances of emotions expressed. Our experiments used static and contextual embeddings (individual and stacked) with Bidirectional Long short-term memory (BiLSTM) and Transformer based models. We occupied rank tenth in the emotion detection task by scoring a Macro F1-Score of 0.2717, validating the efficacy of our implemented approaches for small and imbalanced datasets with mixed categories of target emotions.

heading: 1 Introduction
text: Digitalization and ease of access to internet-based intelligent and interactive technologies have led to an unprecedented amount of textual data generation from social media, customer reviews, and online forums. Therefore, the need to accurately understand and extract emotions and sentiments from text has become imperative for two reasons; first, due to their various crucial applications such as sentiment analysis (Gupta et al., 2023), chatbots, mental health assessment(Wu et al., 2020), social media monitoring, market research, brand management, and customer feedback analysis and second to reduce the human efforts, time and resource requirements. The Shared Task on Empathy Detection, Emotion Classification and Personality Detection in Interactions of WASSA 20231 aims to develop models to predict various targets, including emotion, empathy, personality, and interpersonal-index,
1https://2023.aclweb.org/program/ workshops/
from textual data (Barriere et al., 2023). The shared task consists of five tracks, of which we participated in Track 3: Emotion Classification (EMO), which targets emotion classification at the essay level. This work presents two systems to capture the subtle notion of emotions expressed through texts: a) BiLSTM-based (Graves and Schmidhuber, 2005) DL model using static, contextual, and the combination of static and contextual (stacked) embeddings and b) Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019). Stacked embeddings (Bhandari et al., 2022) are fast-to-train, powerful but underutilized representations; therefore, to reckon their efficacy compared to the transformer model, we have used them in this work. Our proposed systems have performed competitively and got the tenth rank2 in the evaluation phase of the Track 3: EMO task.
The remainder of the paper is structured as follows: Section 2 presents the notable research works on emotion detection. Section 3 presents the problem statement, dataset description, and the preprocessing strategies applied. In section 4, we present our different classification systems and the experimental setup. Section 5 presents the evaluation results of our proposed systems and comparison with other participating teams of the shared task. Finally, section 6 provides the conclusion and discusses the future research directions.

heading: 2 Literature Survey
text: The significance of accurate emotion detection and sentiment analysis extends beyond understanding textual data. Recent research has brought the machines one step closer to mimicking humans’ innate ability to understand emotional cues from and text and different modalities. Works such as (Acheampong et al., 2020; Chatterjee et al., 2019) explore the emotion detection form texts; (Zhong
2The rank is solely based on the submissions done before the deadline of the shared task
581
et al., 2019; Acheampong et al., 2021; Adoma et al., 2020) explores the variants of transformer models useful for emotion detection from texts. Some notable works such as (Wu et al., 2023, 2022; Bostan et al., 2020; Bostan and Klinger, 2018; Buechel and Hahn, 2017; Sosea and Caragea, 2020) have created the novel datasets from textual and conversational settings to address the scarce data challenges in complex domains for emotion detection.

heading: 3 Problem Statement, Dataset Description and Data Preprocessing
text: In this section, we have mentioned the problem statement tackled, the dataset description, and the data-prepossessing techniques implemented for our experiments.

heading: 3.1 Problem Statement
text: In this work, we tackled a multiclass classification problem to predict emotions from essay-level texts. The target labels consist of thirty-one categories of emotions, including individual and mixed sets of emotion categories, as follows: Hope/Sadness, Anger, Sadness, Neutral, Disgust/Sadness, Anger/Disgust, Fear/Sadness, Joy, Hope, Joy/Neutral, Disgust, Neutral/Sadness, Neutral/Surprise, Anger/Neutral, Hope/Neutral, Surprise, Anger/Sadness, Fear, Anger/Joy, Disgust/Fear, Fear/Neutral, Fear/Hope, Joy/Sadness, Anger/Disgust/Sadness, Anger/Surprise, Disgust/Neutral, Anger/Fear, Sadness/Surprise, Disgust/Surprise, Anger/Hope, and Disgust/Hope.

heading: 3.2 Dataset Description
text: The experimental dataset contains long essays of length between 300 and 800 (Omitaomu et al., 2022). The dataset includes news articles and person-level demographic information (empathy, distress, age, race, income, gender, education level, emotion labels, etc.). The dataset was made available as training, development (dev), and test sets where the target labels were shared only for the training and development sets for the evaluation phase. The overall distribution of the dataset is shown in Table 1 and the distribution of each emotion class of the train and dev sets is shown in Table 1.

heading: 3.3 Dataset Preprocessing
text: As evident from Figure 1, the dataset is small and imbalanced and several emotion categories have
only one data point. Also, the mixed categories of emotions in the target class made the task more challenging. Therefore, to overcome these constraints, we have applied a tailored preprocessing strategy along with standard NLP techniques to prepare the input dataset (Dessì et al., 2020; Kumar et al., 2021; Uysal and Gunal, 2014). The preprocessing steps are as follows. The input texts are converted to lowercase to make the dataset uniform in terms of representation (e.g., Emotion and emotion are represented by a common token, emotion). Punctuation, stopwords, newlines, whitespaces, and extra spaces are removed from the text. We have removed the special characters, symbols, and elements which are not part of the standard English language. We have expanded the contractions such as didn’t –> did not. We performed stemming and lemmatization alternatively for experiments but observed a slight decline in the model’s performance. Therefore, we have not considered them for preprocessing the input dataset for the final submission of Track 3: EMO shared task.

heading: 4 Methodology
text: This section describes our different systems (classification models) based on the BiLSTM and transformer model implemented for the emotion classification task.

heading: 4.1 BiLSTM Based DL Model
text: Our first system is a DL model using two BiLSTM layers. More precisely, this model’s architecture consists of an embedding layer, followed by two BiLSTM layers, a dense layer, and an output layer at the end for the multi-class classification. The embedding layer is initialized by input_dim (size of the vocabulary); output_dim: (word vector length), embedding matrix, and sequences length. For ease of understanding, we have summed up the parameters and combination of embeddings used for our experiments in Table 2.

heading: 4.2 BERT
text: The second system is a transformer-based model BERT, created using Keras3 and Tensorflow4. Our BERT model comprises two input layers, a BERT model layer, and two dense layers of 768 embedding dimensions with the Adam optimizer. The parameters used for fine-tuning the model are listed in Table 2.

heading: 4.3 Features Representation
text: We have used pre-trained static and contextual word embeddings for our experiments to generate the feature vectors discussed below.
GloVE (Global Vectors for Word Representation): GloVE5 is an unsupervised learning algorithm that generates word embeddings as dense vector representations of words in a high-dimensional space. It leverages co-occurrence statistics from a large text corpus to capture semantic relationships between words. GloVe embeddings are trained by factorizing a matrix representing the word cooccurrence statistics (Pennington et al., 2014).
fastText: fastText6 was developed by Facebook’s AI Research (FAIR) team (Bojanowski et al., 2017; Joulin et al., 2016). fastText extends the traditional word embeddings by representing each word as a bag of character n-grams, where n can range from 1 to a maximum specified length. This approach allows fastText to capture morphological information and handle out-of-vocabulary words effectively.
BERT: BERT7 embeddings are a type of word
3https://keras.io/ 4https://t fhub.dev/google/collections/bert 5https://nlp.stanford.edu/projects/glove/ 6https://fasttext.cc/ 7https://pypi.org/project/bert-embedding/
representation that captures contextual information in the text. Unlike traditional word embeddings like Word2Vec or GloVe, BERT embeddings take into account the surrounding words when representing a word. This means that the meaning of a word can vary depending on its context.
FLAIR (FastText and Language-Independent Representations): FLAIR8 embedding is a stateof-the-art word representation model that captures contextual information and word semantics by combining the strengths of two powerful techniques: FastText and contextual string embeddings. By combining these techniques, FLAIR embedding provides a robust and language-independent representation of words. It considers both the local context of a word and its global context within a sentence or document (Akbik et al., 2019).

heading: 5 Results and Analysis
text: Table 3 presents the results of our two systems. The BERT base system has significantly outperformed the BiLSTM-based system using the combination of GloVe, fastText and BERT embeddings. Therefore, we have submitted the BERT base result for the shared task evaluation phase. The evaluation of Track 3: EMO shared task is based on the macro F1-score and Micro Jaccard-score, Micro F1-score, Micro Precision, Micro Recall, Macro Precision and Macro Recall are supporting metrics. Our BERT base system has achieved a Macro F1-score of 2.717 and stood tenth9 among participants. Table 4 presents the official results of all the qualifying teams.
8https://pypi.org/project/flair/ 9These rankings were provided by the shared task organiz-
ing team on 11/05/2023.

heading: 6 Conclusion
text: Our system, VISU, participated in the shared task Track 3: EMO of emotion classification tasks of the WASSA 2023, and our BERT base system scored tenth rank. Our experiments conclude that although FLAIR are powerful word representations built to capture out-of-vocabulary words, they are not as
effective as contextual embeddings when used for small and imbalanced datasets. Our future research aims to address the data imbalance and scarce data challenges (Kumar. et al., 2023) by incorporating novel augmentation techniques of domain adaptation(Kumar et al., 2022) to interpret better the emotions expressed in text.

heading: Future Work
text: Our future research aims to address the data imbalance and scarce data challenges (Kumar. et al. , 2023) by incorporating novel augmentation techniques of domain adaptation(Kumar et al. , 2022) to interpret better the emotions expressed in text