SlowGuess/NAACL_2022_OCR huggingface dataset:
Wongyu Kim $^{1}$ , Youbin Ahn $^{2}$ , Donghyun Kim $^{2}$ , and Kyong-Ho Lee $^{2}$
$^{1}$ Department of Artificial Intelligence,  $^{2}$ Department of Computer Science
Yonsei University, Seoul, Republic of Korea
{rladnjsrb9999, ybahn, dhkim92, khlee98}@yonsei.ac.kr
Each utterance in multi-turn empathetic dialogues has features such as emotion, keywords, and utterance-level meaning. Feature transitions between utterances occur naturally. However, existing approaches fail to perceive the transitions because they extract features for the context at the coarse-grained level. To solve the above issue, we propose a novel approach of recognizing feature transitions between utterances, which helps understand the dialogue flow and better grasp the features of utterance that needs attention. Also, we introduce a response generation strategy to help focus on emotion and keywords related to appropriate features when generating responses. Experimental results show that our approach outperforms baselines and especially, achieves significant improvements on multi-turn dialogues.
Humans have empathy which is the ability to understand situations others have experienced and emotions they have felt from the situations (Eisenberg and Strayer, 1987). That ability also enables to interest and console others while sharing a conversation. Thus, empathetic response generation task has been considered noteworthy. Figure 1 shows an example of a multi-turn empathetic dialogue dataset, EmpatheticDialogues (Rashkin et al., 2019) constructed to solve the task. A speaker talks about one of 32 emotion labels and a situation related to the emotion label, and a listener empathizes, responding to the speaker. Existing approaches (Rashkin et al., 2019; Lin et al., 2019; Majumder et al., 2020; Li et al., 2020; Kim et al., 2021) for the task achieve promising results but show limitations when dialogues become long because they extract features from the concatenation of all tokens in the context at the coarse-grained level.
However, at the fine-grained level, each utterance in multi-turn empathetic dialogues has features such as emotion, keywords that each denote
Figure 1: An example of EmpatheticDialogues with response A and B. Response B is from one of state-of-the-art models. Highlighted words are keywords.
what an interlocutor feels and primarily says, and utterance-level meaning that can be known when looking at the entire utterance. In addition, it is a natural phenomenon that features of each utterance differ from the previous, as the dialogue is prolonged. Hence, we humans instinctively recognize these feature transitions, which helps us understand how the dialogue flows and grasp the features of utterance that needs attention. Also, humans respond to others, focusing on emotion and keywords related to appropriate features. Take the example in Figure 1. In the first turn, the speaker is excited to see the speaker's sister in a long time by mentioning keywords (e.g., 'sister', 'visit', 'decade'), and the listener reacts to the excitement and asks about her by mentioning keywords (e.g., 'exciting', 'see', 'live'). However, in the second speaker utterance, the speaker becomes embarrassed because of the speaker's boyfriend's bad table manners by mentioning keywords (e.g., 'boyfriend', 'loud', 'eater'). We humans recognize that the features of second speaker utterance have changed compared to those of previous utterances, and usually decide to be attentive to the features of the second utterance. Then, by focusing on information such as keywords of that utterance and emotion and keywords (e.g., 'bad', 'impression') related to the features of that utterance, humans generate empathetic, coherent,
Human Language Technologies, pages 4118 - 4128
and non-generic responses like response A. However, the model which produces non-empathetic and incoherent response like response B, considers that the features of the first speaker utterance represent the context from the coarse-grained view.
In this paper, we first propose to annotate features on each utterance at the fine-grained level (§4). Then, we introduce a novel Empathetic response generator based on Recognizing Feature Transitions (Emp-RFT), which has two essential parts: Feature Transition Recognizer and Response Generation Strategy. The first part recognizes feature transitions between utterances, utilizing comparison functions of Wang and Jiang (2017), which makes Emp-RFT understand the dialogue flow and grasp appropriate features of utterance that needs attention. The second part helps Emp-RFT focus on emotion and keywords related to appropriate features. Specifically, by fusing context with keywords, such keywords are emphasized within each utterance and get more attention when generating responses. Then, Emp-RFT detects next emotion and keywords that denote emotion and keywords of the response, which helps figure out proper emotion and keywords for generation. Lastly, inspired by Dathathri et al. (2020); Chen et al. (2020), a new mechanism of Plug and Play Language Model(PPLM), contrastive PPLM using contrastive loss, is introduced, which controls Emp-RFT to actively use the keywords detected to be next keywords when generating responses.
We conduct experiments on EmpatheticDialogues. Emp-RFT outperforms strong baselines, particularly, when dialogues are multi-turn.
Our main contributions are as follows. (1) We introduce a novel approach that recognizes feature transitions between utterances, which results in understanding how the dialogue flows and grasping the features of utterance that the model should be attentive to. (2) We propose a response generation strategy including fusing context with keywords, next emotion and keywords detection, and contrastive PPLM. The strategy makes our model focus on emotion and keywords related to appropriate features when generating responses. (3) In the experiments, Emp-RFT outperforms baselines, especially, when dialogues are prolonged.
Since Rashkin et al. (2019) release EmpatheticDialogues, many approaches have been proposed to
generate empathetic responses. Lin et al. (2019) propose mixture of emotional experts. Majumder et al. (2020) propose emotion grouping, emotion mimicry, and stochastic sampling. Li et al. (2020) extract emotional words through lexicon and propose an adversarial generative model. Shen et al. (2021) apply dual-learning with unpaired data for the bidirectional empathy. Gao et al. (2021) integrate emotion cause into response generation process through gated mechanism. Sabour et al. (2021); Li et al. (2022) use implicit commonsense for context modelling. Kim et al. (2021) train a model to extract words that cause the speaker's emotion and attach RSA Framework (Frank and Goodman, 2012) to any generative models to generate responses, focusing on emotion cause words.
Recently, many studies have shown remarkable improvements through recognizing transitions of features between utterances in open-domain multi-turn dialogues. Qiu et al. (2020) perceive transitions of emotion states for context modelling. Zou et al. (2021) propose a module to manage keyword transitions. Zhan et al. (2021) model external knowledge transitions to select a knowledge used for generation. In multi-turn empathetic dialogues, we consider emotions, keywords, and utterance-level meaning (Gu et al., 2021) as important features of each utterance and propose a novel approach of recognizing feature transitions between utterances.
Given context  $con = [u^1, \ldots, u^{n-1}]$ , where an utterance  $u^i = [u_1^i, \ldots, u_{|u^i|}^i]$  consists of  $|u^i|$  words, we can obtain  $e = [e^1, \ldots, e^{n-1}]$  and  $k = [k^1, \ldots, k^{n-1}]$ , where  $e^i$  and  $k^i = [k_1^i, \ldots, k_{|k^i|}^i]$  each denote emotion and  $|k^i|$  keywords of  $u^i$  through data preparation (§4). To conduct next keywords detection, we construct keyword pairs  $kps$  (§4.2) whose each pair has two keywords each from keywords of the speaker utterance and keywords of the listener utterance in the same turn. Finally, given  $con, e, k,$  and  $kps$ , we detect next emotion  $y^y$  and next keywords  $k^y = [k_1^y, \ldots, k_{|k^y|}^y]$ , and generate an empathetic response  $y = [y_1, \ldots, y_m]$ .
4 Data Preparataion
In this section, we introduce feature annotation in the speaker and listener utterances.
Feature	Top-1 Acc	Top-5 Acc	macro-F1
EofSU	46.77	81.26	43.55
EofLU	58.44	89.96	53.25
Feature	TL-P	TL-R	TL-F1
KofSU	41.53	66.58	51.15
KofLU	52.31	60.97	56.30
Table 1: Performances of feature annotations. When we evaluate annotations of EofSU/KofSU, EMOCAUSE (Kim et al., 2021) made based on EmpatheticDialogues, is used to verify emotion and emotion cause words detection models. For evaluations of keyword annotations, we use the metrics, Token-Level(TL) Precision(P), Recall(R), and F1 (DeYoung et al., 2020), usually used in token extraction tasks.
4.1 Feature Annotation in Speaker Utterances
Emotion and Keywords of Speaker Utterance (EofSU/KofSU). Speakers try to say an emotional experience that causes a certain emotion in the utterance. Thus, we leverage a model (Kim et al., 2021) which is trained to jointly detect an emotion and emotion cause words of the speaker utterance, using EmpatheticDialogues. We regard top-6 emotion cause words as keywords and remove stopwords and punctuation in keywords.
4.2 Feature Annotation in Listener Utterances
Emotion of Listener Utterance (EofLU). We fine-tune RoBERTa (Liu et al., 2019) to detect an emotion given a situation description in EmpatheticDialogues. Then, the model predicts an emotion of the listener utterance.
Keywords of Listener Utterance (KofLU). Listeners express empathy in the utterance through three Communication Mechanisms (CMs) (Sharma et al., 2020) including emotional reaction, interpretation, and exploration. Thus, three models are leveraged, where each model is trained to detect words that cause one of three CMs, using another dialogue dataset for mental health support<sup>1</sup>. Then, three models predict such words in the listener utterance. Since predicted words take up slightly a lot in the listener utterance, these words are filtered out in the keyword pairs construction.
Keyword Pairs Construction. Inspired by Zou et al. (2021), keyword pairs  $kps$  are constructed not only to filter out above predicted words, but also to conduct next keyword detection. Given a dialogue
Figure 2: Overall architecture of Emp-RFT.
corpus, all pairs are extracted, where each pair has a head word and a tail word each from keywords in the speaker utterance and predicted words in the listener utterance in the same turn. Then, all pairs are filtered out to obtain high-frequency pairs through pointwise mutual information (PMI) $^2$  (Church and Hanks, 1990) which can measure the association between two words in a corpus. Filtered pairs become kps. A tail word of a  $kp$  is regarded as a keyword of the listener utterances joined to extract that keyword pair.
Performances of feature annotations are summarized in Table 1 and show reliable results. However, test sets for KofLU based on Empathetic-Dialogues, don't exist. Thus, we randomly sample 100 test dialogues in EmpatheticDialogues and ask 3 human workers to annotate whether each word plays an important role for empathizing in the listener utterances. By majority voting, the final verdict on each annotation is decided. We compute the inter-annotator agreement on annotation of test sets for KofLU through Fleiss' kappa  $(\kappa)$  (Fleiss and Cohen, 1973), and result in 0.55, where  $0.4 < \kappa < 0.6$  indicates moderate agreement.
5 The Emp-RFT Model
In this section, we detail Emp-RFT whose overall architecture is shown in Figure 2.
5.1 Context Encoding
Word-Level Encoder. Emp-RFT contains an encoder  $f_{\theta}(\cdot)$  which has the six-layer encoder of BART (Lewis et al., 2020) as the backbone and extracts feature vectors of each  $u^{i}$ . Inspired by BERT (Devlin et al., 2019), we prefix each utterance with a [SEN] token, so  $u_{0}^{i} = [SEN]$ . Then, each token
2We use pairs whose  $\mathrm{PMI}\geq 1$  . The pairs whose tail words are stopwords or punctuation, are removed.
${}^{1}$  A dialogue has a (post, response) pair, and words which cause each CM are annotated on each dialogue.
is represented as  $emb_j^i$ , the sum of the following four embeddings: word embedding, position embedding, role embedding and emotion embedding  $M_e \in \mathbb{R}^{n_{emo} \times d}$ . Then, the encoder transforms each utterance into a list of output hidden states:
$$
[ \hat {h} _ {0} ^ {i}, \dots , \hat {h} _ {| u ^ {i} |} ^ {i} ] = f _ {\theta} ([ e m b _ {0} ^ {i}, \dots , e m b _ {| u ^ {i} |} ^ {i} ]), \tag {1}
$$
where  $\hat{h}_j^i\in \mathbb{R}^d$  . For each utterance, we can obtain utterance-level meaning vector  $\hat{h}_0^i$  derived from the token [SEN], concatenated keyword vectors  $\hat{k}^i\in \mathbb{R}^{|k^i|\times d}$  derived from the tokens corresponding to  $k_{p}^{i}$  ( $p$  is the index for keywords.), and emotion vector  $\hat{e}^i = M_e\hat{h}_0^i$
Feature Transition Recognizer. Emp-RFT has a component that operates as the process illustrated in Figure 3. The component computes feature transition information between feature vectors, utilizing two comparison functions, subtraction and multiplication of Wang and Jiang (2017). Each feature vector is compared to previous two feature vectors  ${}^{4}$  . First,emotion transition information  ${et}{t}^{i}$  is computed:
$$
e t i ^ {i} = \operatorname {R e L U} \left(W _ {e t i} \left(f _ {\text {c o m}} \left(\hat {e} ^ {i}, \hat {e} ^ {i - 1}, \hat {e} ^ {i - 2}\right)\right)\right), \tag {2}
$$
$$
\begin{array}{l} f _ {c o m} \left(\hat {e} ^ {i}, \hat {e} ^ {i - 1}, \hat {e} ^ {i - 2}\right) \\ = \left[ \begin{array}{c} \left(\hat {e} ^ {i} - \hat {e} ^ {i - 1}\right) \odot \left(\hat {e} ^ {i} - \hat {e} ^ {i - 1}\right) \\ \hat {e} ^ {i} \odot \hat {e} ^ {i - 1} \\ \left(\hat {e} ^ {i} - \hat {e} ^ {i - 2}\right) \odot \left(\hat {e} ^ {i} - \hat {e} ^ {i - 2}\right) \\ \hat {e} ^ {i} \odot \hat {e} ^ {i - 2} \end{array} \right], \tag {3} \\ \end{array}
$$
where  $f_{\text{com}}$  and  $\odot$  each denote our transition information computing function and Hadamar product, and  $W_{\text{eti}} \in \mathbb{R}^{d \times 4n_{\text{emo}}}$ . Next, utterance-level meaning transition information  $\text{ut}i^i$  is computed:
$$
u t i ^ {i} = \operatorname {R e L U} \left(W _ {u t i} \left(f _ {c o m} \left(\hat {h} _ {0} ^ {i}, \hat {h} _ {0} ^ {i - 1}, \hat {h} _ {0} ^ {i - 2}\right)\right)\right), \tag {4}
$$
where  $W_{uti} \in \mathbb{R}^{d \times 4d}$ . We then obtain enhanced utterance vector of each utterance by integrating utterance-level meaning vector, and emotion and utterance-level meaning transition information:
$$
\bar {h} ^ {i} = \operatorname {F C} _ {u t t} \left(\left[ \hat {h} _ {0} ^ {i}; e t i ^ {i}; u t i ^ {i} \right]\right), \tag {5}
$$
where  $FC_{utt}$  is a fully-connected layer with size of  $d$ . In addition, keyword transition information
Figure 3: Operation process of feature transition recognizer.
$k t i^{i}$  is computed between concatenated keyword vectors and cross-encoded vectors  $c^t$  , where  $t\in$ $\{i - 1,i - 2\}$  ..
$$
k t i ^ {i} = \operatorname {R e L U} \left(W _ {k t i} \left(f _ {\text {c o m}} \left(\hat {k} ^ {i}, c ^ {i - 1}, c ^ {i - 2}\right)\right) ^ {T}\right), \tag {6}
$$
$$
c ^ {t} = \operatorname {s o f t m a x} \left(Q ^ {i} \left(K ^ {t}\right) ^ {T}\right) \hat {k} ^ {t}, \tag {7}
$$
$$
Q ^ {i} = \hat {k} ^ {i} W _ {Q}, K ^ {t} = \hat {k} ^ {t} W _ {K}, \tag {8}
$$
where  $W_{kti}\in \mathbb{R}^{d\times 4d},W_Q$  and  $W_{K}\in \mathbb{R}^{d\times d}$  .We can obtain enhanced keyword vector of each keyword by integrating keyword vector, and keyword transition information:
$$
\bar {k} _ {p} ^ {i} = \mathrm {F C} _ {\text {k e y}} ([ \hat {k} _ {p} ^ {i}; k t i _ {p} ^ {i} ]), \tag {9}
$$
where  $FC_{key}$  is a fully-connected layer with size of  $d$ . Consequently, the enhanced feature vectors guide Emp-RFT to accurately grasp the features of utterance that the model should be attentive to when given feature transition information.
Utterance-Level Encoder. Emp-RFT contains another encoder  $g_{\phi}(\cdot)$  which has the six-layer encoder of BART, and transforms enhanced utterance vectors with global position embeddings (GPE) into a context representation to capture relationships between utterances (Gu et al., 2021):
$$
[ \ddot {h} ^ {1}, \dots , \ddot {h} ^ {n - 1} ] = g _ {\phi} ([ \bar {h} ^ {1}, \dots , \bar {h} ^ {n - 1} ]). \tag {10}
$$
Emp-RFT consists of hierarchical structures of encoders through word-level and utterance-level encoders. This structure makes Emp-RFT comprehend each utterance at the fine-grained level, and
${}^{3}j,d$  ,and  ${n}_{emo}$  each denote the index for words,hidden size,and the number of emotion classes. The role and emotion embeddings are each for distinguishing two interlocutors and for incorporating the emotion into each utterance.
4If there aren't previous feature vectors, we can obtain those by regarding first output hidden state of a padded utterance as utterance and keyword vectors (Qiu et al., 2020).
understand the context by integrating information based on comprehension of each utterance.
Fusing Context with Keywords. Emp-RFT fuses context with keywords as the process illustrated in Figure 4. We first dynamically build keyword graph for each context. Keywords in each context become nodes and are initialized by corresponding enhanced keyword vectors with GPE. Edges are built across the below cases: (1) between two keywords from the same utterance and (2) between a keyword from a certain utterance and another keyword from the previous two utterances. Also, a tail word in a  $kp$  whose head word is  $k_{p}^{n-1}$  is appended as a node and connected with  $k_{p}^{n-1}$  node. Appended nodes (ANs) are initialized through BART decoder whose parameters are frozen with GPE, and used for next keywords detection. To obtain keyword representation  $\hat{v}_{o}^{i}$  from the keyword graph(o is the index for nodes.), nodes are updated based on multi-head graph-attention mechanism (Velicković et al., 2018; Li et al., 2022). This mechanism makes Emp-RFT not only capture relationships between nodes but also manage influences of each appended node through attention architecture:
$$
\hat {v} _ {o} ^ {i} = v _ {o} ^ {i} + \left| \right| \sum_ {m h = 1} ^ {M H} \alpha_ {o z} ^ {i, m h} \left(W _ {v} ^ {m h} v _ {z}\right), \tag {11}
$$
$$
\alpha_ {o z} ^ {i, m h} = \frac {\exp \left(\left(W _ {q} ^ {m h} v _ {o} ^ {i}\right) ^ {T} W _ {k e y} ^ {m h} v _ {z}\right)}{\sum_ {s \in \mathcal {A} _ {o} ^ {i}} \exp \left(\left(W _ {q} ^ {m h} v _ {o} ^ {i}\right) ^ {T} W _ {k e y} ^ {m h} v _ {s}\right)}, \tag {12}
$$
where  $v_{o}^{i},\parallel ,A_{o}^{i}$ , and  $\alpha_{oz}^{i,mh}$  each denote a node representation, the concatenation of  $MH$  attention heads, the neighbours of  $v_{o}^{i}$  in the adjacency matrix  $A$ , and self-attention weight and  $W_{v}^{mh}$ ,  $W_{q}^{mh}$ ,  $W_{key}^{mh}\in \mathbb{R}^{d_{mh}\times d}(d_{mh} = d / MH)$ . Lastly, we can obtain the fused context representation  $H = [h^1,\dots,h^{n - 1}]$  by fusing the context representation with the sum of keyword representations:
$$
h ^ {i} = \mathrm {F C} _ {f u s e} ([ \ddot {h} ^ {i}; \operatorname {s u m} ([ \hat {v} _ {1} ^ {i}, \dots , \hat {v} _ {| k ^ {i} |} ^ {i} ]) ]), \tag {13}
$$
where  $FC_{\text{use}}$  is a fully-connected layer with size of  $d$ . Consequently, keywords are emphasized within each utterance and get greater attention when generating responses.
Next Emotion and Keywords Detection. EmpRFT detects next emotion  $e^y$  and keywords  $k^y$ , which helps figure out proper emotion and keywords for generation. First, based on the max-pooled fused context representation, next emotion
Figure 4: Operation process of fusing context with keywords.  $AN_{o}$  is the appended node. Some symbols and edges are omitted for simplicity.
distribution is predicted:
$$
P _ {e} = \operatorname {s o f t m a x} \left(M _ {e} \mathrm {M P} (H)\right), \tag {14}
$$
where MP denotes maxpooling. We use the emotion with the highest probability  $(\hat{e}^y)$  for generation. Also, Emp-RFT predicts whether the word of each  $AN$  belongs to the next keywords through the binary classification, where the true label denotes the word belongs to:
$$
P _ {k} = \prod_ {o = 1} ^ {| A N s |} \operatorname {s o f t m a x} \left(W _ {A N} \left[ \hat {v} _ {o} ^ {n}; \mathrm {M P} (H) \right]\right), \tag {15}
$$
where  $W_{AN} \in \mathbb{R}^{2 \times 2d}$ . We consider the words of ANs whose probabilities for the true label  $\geq 0.8$  as the keywords  $(\hat{k}^y)$  for generation.
5.2 Response Generation
Response Generator. Emp-RFT includes a response generator (RG) which has the six-layer decoder of BART as the backbone. Through the four embeddings with  $\hat{e}^y$ , explained previously, we can obtain the input sequence embedding for RG. We prefix it with the sum of node representations corresponding to  $\hat{k}^y$ . Then, RG is fed to predict probability distribution on each next token  $y_{t}$  based on the fused context representation:
$$
P (y | \text {c o n}, e, k, k p s) = \prod_ {t = 1} ^ {m} P \left(y _ {t} \mid y _ {<   t}, H\right). \tag {16}
$$
Training. We apply cross-entropy loss to three objectives (eq. 14, 15, 16), and train parameters of
Emp-RFT in end-to-end manner through the sum of all losses.
Contrastive PPLM. Analysis on the generated responses of the trained Emp-RFT shows that an active reflection of  $\hat{k}^y$  is demanded. Thus, inspired by Dathathri et al. (2020); Chen et al. (2020), we propose Contrastive PPLM with a discriminator using contrastive loss. Existing discriminators (Dathathri et al., 2020; Majumder et al., 2021) are trained to predict whether a sentence contains a certain attribute, using cross-entropy loss. Then, the gradient of the loss is passed to the generative model to generate a sentence containing such attribute during inference. However, since keywords are not attributes but objects, we train a discriminator to predict whether a response in EmpatheticDialogues is more similar to the keyword set of the response(positive sample) than the keyword sets of another responses(negative samples) in the same batch, using contrastive loss based on the similarity between objects:
$$
L _ {p p l m} ^ {a} = - \log \frac {\exp \left(r _ {a} ^ {T} k s _ {a} / \tau\right)}{\sum_ {b = 1} ^ {B} \exp \left(r _ {a} ^ {T} k s _ {b} / \tau\right)}, \tag {17}
$$
where  $r, ks, \tau$  and  $B$  each denote response and keyword set representations, a temperature parameter and batchsize. During inference, we repeatedly sample three random ANs except for nodes of  $\hat{k}^y$ , and consider the sum of such AN representations as one of negative samples and the sum of node representations corresponding to  $\hat{k}^y$  as a positive sample. Then, the gradient of the contrastive loss is passed to Emp-RFT.
6.1 Dataset and Baselines
Dataset. Experiments were conducted on EmpatheticDialogues (Rashkin et al., 2019) which contains 24,850 multi-turn dialogues. For each dialogue, we can extract a certain number of instances corresponding to the number of turns within the dialogue. This totals to 47,611 instances, where 22,761 are multi-turn. In one turn of a dialogue, a speaker talks about one of 32 evenly distributed emotion labels and a situation related to the emotion label and a listener empathizes by responding to the speaker. Following the instructions of the dataset, we use 8:1:1 train/valid/test split.
Baselines. We compared Emp-RFT to the following five baseline models: (1) MoEL (Lin et al., 2019) is a transformer-based generative model,