SEC: Context-Aware Metric Learning for Efficient Emotion Recognition in Conversation (2024)

Abstract: The advent of deep learning models has made a considerable contribution to the achievement of Emotion Recognition in Conversation (ERC). However, this task still remains an important challenge due to the plurality and subjectivity of human emotions. Previous work on ERC provides predictive models using mostly graphbased conversation representations. In this work, we propose a way to model the conversational context that we incorporate into a metric learning training strategy, with a two-step process. This allows us to perform ERC in a flexible classification scenario and end up with a lightweight yet efficient model. Using metric learning through a Siamese Network architecture, we achieve 57.71 in macro F1 score for emotion classification in conversation on DailyDialog dataset, which outperforms the related work. This state-of-the-art result is promising in terms of the use of metric learning for emotion recognition, yet perfectible compared to the micro F1 score obtained.

heading: 1 Introduction
text: Computer Mediated Communication (CMC) is constantly evolving and new means of communicating are emerging. With the advent of conversational agents, there is a need to detect emotions within a conversation. Although many modalities are now considered in the communication process, the textual modality still remains essential for fast and easy everyday communication, through messaging applications, social media, and other networking platforms. Textual modality, however, is ambiguous, it does not preserve the extra-linguistic context, especially for dyadic human-to-human conversations. One main ambiguity that arises in CMC is the emotional state of the speaker, often misinterpreted by humans through short, and unpolished messages. This motivates Emotion Recognition in Conversation (ERC), a trending research topic
dedicated not only to identifying emotion in messages, but also on taking into account the conversational context to recognize emotions. ERC has been shown to be challenging, especially with respect to the way to represent the context (Ghosal et al., 2021). Lately, it has seen a surge of multimodal models (Wen et al., 2023; Liang et al., 2023; Fan et al., 2024) and graph-related approaches (Zhang et al., 2023; Wang et al., 2023; Li et al., 2023) which often try to map the pattern of each speaker and better represent the conversational context, often resulting in good performance at the cost of efficiency. One additional issue ERC models are facing is their dependency on labels, models are mainly supervised and face the issue of extreme label imbalance due to emotional utterances being so scarce.
In this paper, we tackle these two challenges by incorporating the conversational context into metric learning while heavily controlling the data imbalance by multiple means. Considering that we want to tackle information across emotions to make our model usable for variants of emotions that go beyond the scope of the 6 basic emotions, we do not use supervised contrastive learning (Khosla et al., 2020) in our method. Instead, we focus on a twostep process to update the model using both direct label predictions through a cross-entropy loss and relative label assignment through the contrastive loss. This two-step process is quite straightforward, while using isolated elements, such as isolated utterances. However, to the best of our knowledge, contextual representation through contrastive learning for ERC has yet to be used. This represents our main contribution in this paper, as we present a model that can achieve competitive performance compared to the state-of-the-art while rendering the adaptation to other emotion labels feasible. Thus, our model can be applied and adapted in multiple contexts that require recognition of different label granularities.
11
Our main contribution lies in the development of a metric-learning training strategy for emotion recognition in utterances that incorporates the conversational context. The presented model leverages sentence embeddings and Transformer encoder layers (Vaswani et al., 2017; Devlin et al., 2019) to represent dialogue utterances and deploy attention on the conversational context. Our method involves Siamese Networks (Koch et al., 2015) in the setup but can be adapted to any metric-learning model. We further demonstrate that our approach outperforms some of the latest state-of-the-art Large Language Models (LLMs) such as light versions of Falcon (Penedo et al., 2023) or LLaMA 2 (Touvron et al., 2023). In addition, our method is efficient in the sense that it involves lightweight, adaptable and quickly trainable models, which still yield state-ofthe-art performance on the DailyDialog dataset in macro F1 score with 57.71% and satisfactory results on micro F1 with 57.75%.
Our code and models are available on GitHub 1 to reproduce training, inference and qualitative experiments.

heading: 2 Related Work
text: ERC. Although most of the studies on ERC have been carried out on multimodal datasets (Song et al., 2022; Li et al., 2022; Hu et al., 2022), thus leveraging multi-modality, there are still some models developed for emotion recognition on textual conversation only, whether it be on multimodal datasets restricted to text such as IEMOCAP (Busso et al., 2008) or MELD (Poria et al., 2019), or on a fully textual dataset such as DailyDialog (Li et al., 2017). The advent of deep learning enables significant progress in ERC on text, starting by the use of Recurrent Neural Networks (RNN) (Rumelhart et al., 1985; Jordan, 1986) by Poria et al. (2017). Further work using recurring structures followed, such as DialogueRNN (Majumder et al., 2019; Ghosal et al., 2020). This model leverages the attention mechanism (Bahdanau et al., 2014) combined with RNN. Graphbased methods also proved to be efficient as shown in (Ghosal et al., 2019), not only as such but also when considering external knowledge, as Lee and Choi (2021) use a Graph Convolutional Network (GCN) to perform ERC by extracting relations between dialogue instances.
Existing work on ERC relies mainly on evaluat-
1https://github.com/B-Gendron/sentEmoContext
ing their model using a micro F1 score excluding the majority neutral label. However, recent work actually skipped this evaluation to instead focus only on the macro version of this metric (Pereira et al., 2023), while other considered the Matthew Coefficient Correlation as an indication suitable for this task (Guibon et al., 2021).
In this work, we focus on DailyDialog, which consists of artificially human-generated conversations about daily life concerns, with utterance-wise emotion labeling. Liang et al. (2022) propose a model based on Graph Neural Networks (GNN) and Conditional Random Fields (Lafferty et al., 2001) (CRF) that achieves 64.01% in micro F1.
Although it is known not to provide the best performance compared to few-shot learning approaches (Dumoulin et al., 2021), meta-learning allows better generalization through more robust training (Finn et al., 2017; Antoniou et al., 2019), which is particularly adapted in the case of emotion detection due to both variability and complexity of human feelings (Plutchik, 2001).
Metric learning. As reviewed by (Hospedales et al., 2022), a meta-learning approach consists in a meta-optimizer that describes meta-learner updates, a meta-representation that stores the acquired knowledge and the meta-objective oriented towards the desired task. This optimization-based meta-learning setup provides end-to-end algorithms often based on episodic scenarios (Ravi and Larochelle, 2016; Finn et al., 2017; Mishra et al., 2017) that reflect the "learning to learn" strategy. Besides, learning to learn implies second order gradient computations which is costly. Palliative solutions to this problem, such as implicit differentiation (Lorraine et al., 2020), still involve a trade-off between performance and memory cost (Hospedales et al., 2022). Therefore, variants has emerged such as metric learning, which meta-objective is to learn the meta-representation itself. Starting with Siamese Networks (Koch et al., 2015), this model structure leverages parameter sharing between identical sub-networks to learn a distance between data samples. Relation Networks (Sung et al., 2018) also consider a distance metric, departing from the traditional Euclidean approach. Matching Networks (Vinyals et al., 2016) leverage training examples to identify weighted nearest neighbors. Prototypical Networks (Snell et al., 2017) compute average class representations and utilize cosine distance for element comparison.
This model has been adapted to perform ERC in a few-shot setting by Guibon et al. (2021) in a way that outperformed few-shot learning baselines.
In this work, we focus on the Siamese Networks architecture. It has the advantage of being conceptually simple, which makes it easily controllable and scalable. Nevertheless, the model structure proposed in this paper is easily adaptable to more complex meta-learning setups. Siamese Networks have been used, for example, in NLP for intention detection on text (Ren and Xue, 2020), in computer vision for facial recognition (Hayale et al., 2023), and in complex representation learning (Jin et al., 2021).

heading: 3 Methodology
text: In this work, we use a metric-learning architecture based on learning emotions as they relate to each other, thus extracting meta-information from the data. The model is a Siamese network (Koch et al., 2015) with three identical sub-networks, whose outputs are compared using the triplet loss (Schultz and Joachims, 2003). Initially applied to computer vision problems (Chechik et al., 2010; Schroff et al., 2015), triplet loss is defined on a triplet of data samples (a, p, n) so that if a and p belong to the same class and n belongs to a different class, then:
L(a, p, n) = max {d(a, p)− d(a, n) + margin, 0}
where the margin parameter is a strictly positive number.
While the triplet loss could be used in several strategies, ranging from only retrieving the most difficult triplets (when the positive is far from the anchor, meanwhile the anchor is close to the negative) to skipping the most easy ones (i.e. when the positive is closer to the anchor), we only tackle the overall strategy by considering each triplet in our data, due to the limited size of the data.
Isolated representations. As the aim of our experiments is to characterize the contribution of conversational context to emotion prediction, we first developed a baseline model on isolated utterances. This formally refers to computing emotion predictions for utterances independently of their context. To do this, we first consider a mapping for each utterance word to its associated FastText embedding (Bojanowski et al., 2017). From such embeddings, aforementioned (a, p, n) triplets are randomly sampled and given as input for the Siamese Network, whose sub-network gradually improves
in emotion prediction as triplet loss backpropagates.
Contextual representations. Regarding the contextual case, we build contextual utterance representations upon a BERT-like encoding. Sentence embeddings are preferred to word-piece embeddings (like BERT produces) as they provide lighter utterance representations. After the dialog is mapped to its associated series of pretrained embeddings, these outputs are concatenated forming a dialog representation, and contextual information is considered by deploying attention over it. Concretely, a Transformer encoder layer is stacked to the gathered frozen pre-trained embeddings. This newly conversation-aware dialog representation is then split at [SEP] tokens to end up with contextual representations at the utterance level, on which the emotion prediction is performed. In order to fit contextual utterance representations to the emotion prediction objective, we add an emotion classifier that is pre-trained on DailyDialog training set. The classifier is not frozen to ensure a complete backpropagation. Meanwhile, contextual representations are optimized according to the metric learning objective, using a triplet loss. The whole training procedure is illustrated in Figure 1. This training scenario enables both individual and relative emotion learning, in such a way that each learning phase strengthens the other. Thanks to this meta-learning setting, meta-information about emotions is extracted, and we can expect that this model is able to achieve relevant classification on unseen labels in a few-shot setting.

heading: 4 Experimental Protocol
text: Data. All the experiments have been carried out on DailyDialog dataset (Li et al., 2017) that provides more than 10,000 dialogues about daily concerns along with utterance-wise emotion labeling. In addition to providing utterance-level emotion labeling, an advantage in using DailyDialog is that it is relatively small, therefore it is quite easy to handle the entries and run tests on it. There exist six emotional labels (anger, disgust, fear, happiness, sadness and surprise) and a neutral label. Regarding emotion prediction, the evaluation is carried out only on the emotional labels following previous work procedure (Ghosal et al., 2021; Zhong et al., 2019). We use the original dataset splits (train, validation and test) from Li et al. (2017). The main characteristics from DailyDialog dataset
are visible in Table 1.
Model specificities. For the isolated utterance model, we consider two different types of subnetworks being simple linear layers and Long Short-Term Memory layers (LSTM) (Hochreiter and Schmidhuber, 1997). In the contextual case, the sub-network is a Transformer encoder fed with sentence embeddings. We carried out experiments with three different models of pre-trained sentence Transformers available in the Python library sentence transformers2: MPNet (Song et al., 2020), MiniLM (Wang et al., 2020) and RoBERTa (Liu et al., 2019). In order to ensure a good balance, the (a, p, n) triplets are made at this stage, meaning right before applying the pretrained emotion classifier, which is composed of a linear layer stacked upon one Transformer encoder layer.
Training specificities. Whether it be for the isolated utterance model or for the contextual one, the emotion prediction is always performed at the utterance level, therefore the triplets are always utterance triplets. This involves balance issues as DailyDialog dataset is very imbalanced regarding
2https://www.sbert.net/
emotion labels (Figure 4). Indeed, the class rebalancing induced by sampling triplets according to a uniform distribution does not sufficiently mitigate bias during training and prevents the loss from converging due to excessive oversampling in frequent classes. Thus, we addressed the imbalance problem all along the training pipeline, by implementing a random sampler weighted with inverse label frequencies to account for the rareness of some emotional labels like fear or disgust.
Evaluation. For quantitative evaluation we needed to account for both performance and relevancy of the training procedure so that generalization abilities enabled by the meta-learning architecture are actually usable. This way, we selected, in addition to usual performance metrics, a more demanding metric: Matthews Correlation Coefficient (MCC) (Cramér, 1946). This measures a Pearson correlation (Pearson, 1895) between the predicted and the actual class, giving more precise information on classification quality than F1 score (Baldi et al., 2000). Using TP . TN . FP and FN as respectively the number of true positives, true negatives, false positives and false negatives, P and R being respectively precision and recall, and N the total number of samples, MCC was originally defined in (Matthews, 1975) as:
MCC = TP/N −R× P√ PR(1−R)(1− P )
(1)
Comparison with LLMs. In order to place the results of our isolated and contextual models into perspective, we compare our models with stateof-the-art LLMs, namely LLaMA (Touvron et al., 2023) and Falcon (Penedo et al., 2023). Both are considered with instruction fine-tuning and evaluated on text generation inference in a zero-shot
setting. We developed a prompt asking for prediction on the last utterance of each DailyDialog test set dialog, regarding the conversational context. For both LLMs, we went through an iterative process to find the most adapted prompt in the sense that the model actually generates only one label. The prompt is the same for each model of the same type (either LLaMA or Falcon). We experienced more difficulty on prompt tuning with Falcon as the model generates happiness on 86% of DailyDialog test set. Both prompts full texts are provided in Figure 2.

heading: 5 Results
text: Table 2 gives an overview of the different results obtained by the research community on ERC with DailyDialog. This actually shows a slow progression since 2017 where Poria et al. (2017) proposed to evaluate the model on the micro F1 score excluding the majority class (i.e., the neutral class). This became the first baseline for this task, achieving 50.24 in micro F1 score. However, the current state-of-the-art model now achieves 64.07 in micro F1 score (Liang et al., 2022) which amounts to a 14 points improvement during 6 years. As visible in Table 2, the community mainly followed this pattern and evaluation scheme. However, in
this paper, we think it is important to also consider the macro F1 score, excluding the majority class, as it shows the overall performance in all emotions. Some work has already decided to do so since 2020 (Ghosal et al., 2020), leading to an improvement of ~2.5 points in 3 years. Following this idea, Figure 3 and Table 3 illustrate this adaptability in emotion prediction showing the detailed classification results.
Compared to these results, our SentEmoContext model achieves 57.75 in micro F1 score, which is a decent but somewhat modest result in terms of metric comparison. However, Table 2 also shows the average performance of our model over 10 runs. Our SentEmoContext is state-of-the-art on the macro F1 score with 57.71 points, outperforming CD-ERC (Pereira et al., 2023) by 6.48 points, which is considerable since they only focused on this metric, and TODKAT (Zhu et al., 2021) by 5.15 points. We also evaluate our model using the multiclass MCC (Matthews, 1975; Baldi et al., 2000) score to ensure that the model does not arbitrary decide. Given an MCC score range of -1 to 1, and 0 indicating randomness, the 0.49 MCC score of the SentEmoContext model indicates that our approach is balanced and accurate in terms of predictions (Chicco and Jurman, 2020). Of course, we cannot compare with other ERC works with the MCC metric, as they did not use it. However, we think it is important to consider it as an additional metric to indicate the quality of the classification, minimizing the effect of the highly imbalanced data from conversations.
Given these results, our SentEmoContext performs really well considering that we only need ~20 minutes per epoch on GPU Nvidia A40 (45 GB
RAM) and train it using only 5 epochs. This makes a striking difference from existing approaches that use multiple streams per speaker (Pereira et al., 2023), graph modeling for the representation of context and knowledge (Zhong et al., 2019; Li et al., 2021), or other heavy representations in their model (Liang et al., 2022). In addition to this, our
model is stable with a standard deviation of only 0.01 on average across the three metrics, which reinforces the quality of such an efficient approach.

heading: 5.1 Comparison with Emotion Classifiers on Utterance Level
text: Table 4 shows the results of the direct emotion classification on utterances. For this task, we only considered the 6 emotion labels, excluding the neutral one not only from the evaluation but also from the training. By doing so, we want to determine the difference between our approach and a dedicated emotion classifier. This also serves as an ablation study for our SentEmoContext model, since this step is part of its training. With Table 4, we can see that our model leverages both the embedded conversational context and the metric learning scheme to increase all metrics. We can especially note the difference in terms of macro F1 scores, which shows the importance of the triplet loss represen-
tation in our model. Indeed, the emotion utterance classifiers are trained using batches balanced on the whole training set distribution and a weighted crossentropy loss. Results show that it is not enough to deal with extreme imbalanced data such as conversations.

heading: 5.2 LLMs Results
text: The LLM results in a zero shot setting are visible in Table 5. These serve as an indication on the performance of such models, albeit in their lightweight version, in the ERC task. Although these generative models are not designed for this quite peculiar task, they still manage to outperform the utterance emotion classifiers of Table 4, which can be considered as a display of emergent capacities of LLMs (Srivastava et al., 2022).
5.3 Imbalance Factor
Although Table 1 shows the characteristics of the dataset, it omits to present the main characteristic of the conversational data in terms of emotion labels: the extreme imbalance. Most of the difficulty in ERC comes from the label definition, the context, but also from the imbalance factor that prevents the model from easily learning the representation of emotions in the context. Figure 4 shows the distribution of the labels in DailyDialog, without the neutral one. Considering the latter is the majority label and is excluded from the evaluation metrics by all the ERC community. The fact that even in the emotion labels the data is that imbalanced proves to be challenging and needs to be addressed. In fact, we are derived from Guibon
et al. (2023) to tackle the imbalance in two steps. First, we balance the data loader to produce more balanced batches given the training set weights. Second, we weight the cross-entropy loss from the emotion classifier considering the remaining imbalance on each batch.
In addition to this, we add another way to address the imbalance. By considering triplets, we remove the imbalance factor while using hidden states that come from balanced representation. We think this partly explains the effectiveness and the efficiency of our model, considering its limited size compared to the related work.

heading: 6 Discussion

heading: 6.1 Model Size and Efficiency
text: Our SentEmoContext is efficient. It produces stateof-the-art results on macro F1 score and good results on micro F1. However, our model trains relatively fast and does not require a lot of epochs to converge. We think this efficiency, along with the limited memory needed to train, is due to both our two-step backpropagation and to the fact that we are using utterance-embedded representations with sentence transformers. Thus, our model can efficiently tackle long conversational contexts with limited memory cost.
In addition, Table 6 shows the difference between the models we used in terms of size, parameters, and number of layers. Our model is relatively small considering the recent advances and related work in ERC, but also compared to LLMs.

heading: 6.2 Relative Label Representation
text: Our approach actually learns twice from the data, first by using a supervised setting, and then by actually considering the relative distances between encoded elements, updating through the triplet loss. This enables the use of our model to different conversation datasets with different labels. The only requirement to extend the scope of this model would be to consider another triplet sampling strategy ignoring labels, such as the batch-hard strategy (Do et al., 2019).

heading: 7 Conclusion
text: In this paper, we present our SentEmoContext model, which comes from an approach that mixes utterance level representation, metric learning, and Siamese Networks. This model efficiently represents the conversational context, which makes
it achieve state-of-the-art macro F1 score with 57.71, and satisfactory micro F1 scores with 57.75 on the Emotion Recognition in Conversation on DailyDialog. We also propose to use the Matthew Correlation Coefficient to better evaluate this task.
With SentEmoContext we use contrastive learning with balanced samplers to minimize the imbalance factor, which is inherent to conversational data. We also leverage sentence BERT to both minimize the memory required for training considering the whole conversational context and to actually represent the conversational context by considering utterances as the minimal unit. This led to a more robust and efficient training method that does not require a lot of epochs to obtain satisfac-
tory results. We also show that small- to averagesize open-source LLMs are still behind on emotion recognition in conversation, as it requires a lot of context to be incorporated in the prompt and is not specifically relevant to generative models.
In our future work, we want to consider applying this approach to other datasets, with added modalities, to stress-test our model. We also plan to use it on slightly different labels, as our model learns relative positions toward labels. Thus, we plan to adapt it to a setting leaning towards meta-learning.

heading: 8 Limitations
text: The first limitation we faced with LLMs is the requirement of high-memory GPUs to test them.
This explains why in Table 5 we only consider the lightweight version of these two open source LLMs. While LLaMA 7b and 13b gave answers in a good format, i.e. with only one label chosen, Falcon did not behave the way we wanted. In order to solve this, we look for the first mentioned emotion in the output to consider it as a label.
Also, it is important to note that we did not want to tackle OpenAI’s ChatGPT due to the fact that we do not have a clear control on the model version, size and approach used behind its API, but also because we wanted to consider open source models, and open source data as we will release both our models and source code to the community. Moreover, we limited ourselves to LLaMA 2 as experiments were performed prior to the release of LLaMA 3.
An additional possible limitation on LLMs is the context size. In ERC, context size is key, but with LLMs adding examples in the prompt to do fewshot learning would take a lot of space in the overall context, the prompt being part of the context. This explains our decision to only consider zero-shot in this paper for LLMs, even though we should also consider prompt tuning to enhance them on this specific task.

heading: Acknowledgments
text: Experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS. RENATER and several Universities as well as other organizations (see https: //www.grid5000.fr).

heading: Future Work
text: In our future work, we want to consider applying this approach to other datasets, with added modalities, to stress-test our model. We also plan to use it on slightly different labels, as our model learns relative positions toward labels. Thus, we plan to adapt it to a setting leaning towards meta-learning