# SAPBERT: Speaker-Aware Pretrained BERT for Emotion Recognition in Conversation
## 1 Introduction  
Advance in artificial intelligence has increased the attention to the empathetic system and emotional interaction between the human and machine, especially in a conversation system such as a chat-bot and virtual assistant. However, it remains a challenge for both the machines and humans to detect an emotion within a conversation. In addition, it is even more difficult when only the conversation text data are available.  
In ERC tasks, there have been many attempts based on deep-learning to understand human emotions with only text data. For better understanding the sequential contexts in conversations, [PERSON] et al. [1] and [PERSON] et al. [2] proposed the RNN-based model. To alleviate the long-term dependency problems in an RNN-based model, a graph neural network (GNN)-based model was introduced [3]. To understand the context of conversations, an external commonsense knowledge was utilized [4; 5; 6].  
Due to the remarkable improvement in pre-training language models in a variety of NLP tasks [7; 8; 9; 10], there have been attempts to use a pre-training model for ERC tasks. [PERSON] et al. [11] suggested that generative conversational models can be leveraged to transfer knowledge for the ERC task, while [PERSON] et al. [12] suggested the ConvCom (Conversation Completion) task as an effective pre-training method for the ERC task.  
One of the unique properties of conversations is the nature of the participating speakers, and the ERC task requires capturing speaker-related contextual information: self-dependency (Figure 1) and inter-speaker dependency (Figure 2). Self-dependency means the aspect of emotional influence that speakers have on themselves, while inter-speaker dependency represents the emotional interaction among speakers during a conversation. Capturing this contextual information can be a main challenge for understanding conversations and addressing the ERC task. In this study, we propose a new pre-trained model,
For model's better understanding of conversations, we aim to enhance model's ability to capture the speaker-dependent contexts, the coherence of conversation and the whole conversation context through introducing three pre-training objectives: 1. Speaker Classification, 2. Masked Utterance Regression, and 3. Last Utterance Generation. We use the output of the SAPBERT's pre-trained conversation-level BERT as the encoder of utterances and perform experiments on two representative ERC datasets, IEMOCAP [13]  
Figure 1: An example of self-dependency context. As shown in the figure, each speaker maintains a consistent emotional state regardless of the counterpart speaker’s emotional state or utterance.  
Figure 2: An example of inter-speaker dependency context. As shown in the figure, Speaker B’s emotional state shift from neutral to frustrated is triggered by Speaker A’s response.  
and MELD [14]. The results show that our approach can be effective for understanding the conversation and can assist with the ERC task. In addition, we perform an ablation study on the pre-training objectives to demonstrate that the proposed approach, especially Speaker Classification, is effective.  
In summary, our contributions are as follows:  
* We present a new pre-training strategy for better understanding of the conversation: 1. Speaker Classification, 2. Masked Utterance Regression, and 3. Last Utterance Generation.
* We demonstrate that our pre-training strategy is effective for understanding conversations and can improve the ERC performance through experiments.