Facilitating Multi-turn Emotional Support Conversation with Positive Emotion Elicitation: A Reinforcement Learning Approach
Year: 2023

Abstract:
Emotional support conversation (ESC) aims to provide emotional support (ES) to improve one’s mental state. Existing works stay at fitting grounded responses and responding strategies (e.g., question), which ignore the effect on ES and lack explicit goals to guide emotional positive transition. To this end, we introduce a new paradigm to formalize multi-turn ESC as a process of positive emotion elicitation. Addressing this task requires finely adjusting the elicitation intensity in ES as the conversation progresses while maintaining conversational goals like coherence. In this paper, we propose SUPPORTER, a mixture-of-expert-based reinforcement learning model, and well design ES and dialogue coherence rewards to guide policy’s learning for responding. Experiments verify the superiority of SUPPORTER in achieving positive emotion elicitation during responding while maintaining conversational goals including coherence.

heading: 1 Introduction
text: Emotional support (ES) aims to reassure a person to recover from emotional distress and improve one’s mental state (Burleson, 2003). It is a manifestation of emotional intelligence in social interactions (Heaney and Israel, 2008; Atoum and Al-Shoboul, 2018). Endowing ES into social dialogue systems for building helpful and trustful agents is an emerging trend (Huang et al., 2020; Rains et al., 2020).
To achieve this goal, a typical practice is modeling empathy, which aims to perceive and understand the situation and feelings of others (Keskin, 2014). Yet, the empathetic conversation (Rashkin et al., 2019) is inherently deficient in providing ES as (1) Lack of consideration of multi-turn conversation. Just making empathetic responses in each single dialogue turn leads to ignoring the user’s feedback and mental state changes in multi-turn
∗Work done during internship at the CoAI Group. †Equal contribution. ‡Corresponding author.
adjusts the intensity of empathy and elicitation to achieve the goal of improving the user’s mental state.
interaction. (2) Lack of awareness of emotional elicitation. Only emanating emotional resonance fails to help users jump out of negative mental states. Although Liu et al. (2021) design emotional support conversation (ESC) task promising to remedy these deficiencies, existing works (Tu et al., 2022; Cheng et al., 2022; Peng et al., 2022) stay at fitting grounded responses and responding strategies (e.g., question) while ignoring the effects of such efforts on ES. They do not fully model the essential working mechanism of ESC and lack explicit goals to guide a user’s emotion to a positive transition in the multi-turn process. Thus, they are still insufficient to lay out an entire ESC process and cannot effectively improve one’s mental state.
To this end, we introduce multi-turn ESC with positive emotion elicitation, a new paradigm aims to progressively empathize and elicit users to reach a better mental state through multi-turn conversation. Addressing this task is challenging (an example is in Figure 1): First, in a realistic multi-turn ESC, the user’s emotions often transit towards positive (e.g., the user’s emotion starts with negative and ends with positive, i.e., “My school was closed”
1714
→ “I feel better now”) with fluctuation (e.g., the user’s negative emotions in the first two turns gradually deepen, i.e., “My school was closed” → “I don’t even know”), which requires the agent to equip with the mechanism dealing with complex situations to respond satisfactorily (Shibata et al., 2014; Yoshino and Kawahara, 2015). Second, for ES, the ES response requires a delicate balance between empathy and elicitation. Only empathizing without eliciting falls into a negative emotional cycle, while the opposite setting brings a sense of distance in communication. They need to be progressively and purposefully adjusted in ongoing interactions, e.g., the agent expresses empathy of varying emotional polarity (negative → negative → positive) and carefully increase the intensity of elicitation (only empathy → weak elicitation → strong elicitation). Third, for language expression, the ES response purposefully elicits positive emotions but should not undermine general conversational goals like coherence. Making an eliciting response that is out of the dialogue context, e.g., replacing “I understand you. I would ... happened to me.” with “Come on! I believe ... find a solution!”, may cause users to resent and block useful feedback.
In this paper, we propose SUPPORTER1 to facilitate multi-turn emotional SUPPORT conversation with positive emotion Elicitation using a mixtureof-expert(MoE) based Reinforcement learning(RL). MoE designs heuristic experts associated with specific tasks to learn diverse semantics by characterizing dialogue context, where: (1) To cope with the user’s emotional fluctuation in the ongoing conversation, experts are devised as positive and negative experts as a whole; (2) To inspire ES of responding, the emotion experts of MoE are designed to predict the user’s emotional states that are possibly transited to; (3) To inspire the expression of responding, the keyword experts of MoE are designed to predict the keywords that maintain the dialogue coherence. With experts as candidates, our RL agent learns conversational semantic encoding policy and purposefully selects experts with expert selection policy for response generation. To achieve the goal of positive emotion elicitation during responding while maintaining conversational goals like coherence, we optimize policy by carefully constructing the rewards: (1) ES rewards consider the conversation progress to dynamically adjust the elicitation
1The project repository is available at https://github. com/jfzhouyoo/Supporter
intensity of positive emotion; (2) Dialogue coherence rewards involve keyword-level and sentencelevel guides to finely maintain coherence.
Our contributions are summarized as follows: (1) We introduce a new paradigm by carefully dissecting the challenges of formalizing multi-turn ESC as a process of positive emotion elicitation.
(2) We propose SUPPORTER, an MoE-based RL model with carefully constructed ES and dialogue coherence rewards, elicits positive emotion during responding while maintaining dialogue coherence.
(3) Extensive experiments show the superiority of SUPPORTER with automatic, interactive human, and novel ES and dialogue coherence evaluations.

heading: 2 Related Work
text: Empathetic Conversation To construct a warm dialogue system, a milestone is to endow it with empathy (Rashkin et al., 2019). Considering affective empathy (Lin et al., 2019; Majumder et al., 2020; Li et al., 2020, 2022), i.e., perceiving the user’s emotion, and cognitive empathy (Zheng et al., 2021; Sabour et al., 2022; Zhou et al., 2022), i.e., understanding the user’s situation, puts the psychological theory of empathy into practice. Limited by focusing on a single-turn empathy and lack of emotional induction, it is difficult to achieve the higher goal of improving the user’s mental state due to failure to help one jump out of the negative situation.
Emotional Support Conversation To remedy above deficiencies, Liu et al. (2021) design ESC for providing ES in interactions. Our work is related to existing works on ESC but differs in task definition as we focus on enhancing the elicitation effect of positive emotion of responses instead of responding strategy prediction (e.g., question) and grounded response generation. Although fusing knowledge (Tu et al., 2022; Peng et al., 2022) and planning strategy (Cheng et al., 2022) are beneficial for wordoverlap metrics (e.g., Bleu), we argue whether the gains serve to ES is opaque and less convincing due to lacking corresponding evaluation mechanisms.
Positive Emotion Elicitation Conversation To free users from emotional distress and advance the conversation towards an optimistic state, positive emotion elicitation is an intuitive solution (Mishara et al., 2007; Jiang et al., 2021). Previous works (Hasegawa et al., 2013; Lubis et al., 2018, 2019a,b) posit the emotional elicitation process as an ideal single-turn dialogue with linear emotional changes
(Wang et al., 2022). However, realistic scenarios often involve multi-turn interactions with complex emotional fluctuations. To weaken the previous strong hypothesis, we extend positive emotion elicitation to ESC by well defining challenges, and take it as a real-world application of the solution.

heading: 3 Preliminaries
text: At the t-th turn of dialogue, given dialogue context Ct = {x1, y1, . . . , xt−1, yt−1, xt}, our goal is to generate the response yt which serves to improve the user’s mental state. To equip this ability, the response generation process should achieve specific goals related to ES and language expression.
ES for Positive Emotion Elicitation Providing effective elicitation during multi-turn ESC suffers from two issues: First, the elicitation intensity of positive emotion needs to be adjusted progressively as the conversation progresses. Maintaining weak elicitation (e.g., “I understand you”) or strong elicitation (e.g., “Come on”) may fail to shake one’s mental state. Second, the elicitation effect of positive emotion needs to be indirectly verified by the feedback from the user’s next turn utterance. It means the elicitation intensity should consider the future fluctuation of the user’s emotional states. In this work, we construct conversation-level and turn-level ES rewards to guide the model’s learning of elicitation policy and conduct corresponding automatic and interactive human evaluations for measuring the ES performance of responding.
Language Expression for Dialogue Coherence The purpose of generative processes to enhance elicitation induces two attendant issues: First, with-
out proper controls may lead to greedily pursuing the goals of elicitation while discarding the contextual coherence, e.g., “Come on!” with strong elicitation as a response in the context of the user continuing to express negative emotions. Second, whether the response meets the user’s expectations needs feedback from the user’s future utterance. It means maintaining coherence with future dialogue is also crucial. In this work, we construct contextual and future dialogue coherence rewards to guide the model’s learning of bi-coherent expressions and perform the automatic and interactive human evaluation of conversational goals including coherence.

heading: 4 Methodology
text: In Figure 2, our SUPPORTER takes dialogue context as input to construct state sequence, which is encoded by a dialogue encoder as the conversational semantic encoding policy. The mixture-of-expert associated with emotion and keyword prediction tasks characterize state semantics to yield action candidates of the expert selection policy, which are purposefully selected for inducing state update. We use the updated state to generate response and further optimize the policy by measuring how well the response reaches the goal of ES and dialogue coherence with the well-designed parallel rewards.

heading: 4.1 Multi-task Mixture-of-Expert
text: As a key component of SUPPORTER, we first introduce the structure of multi-task mixture-of-expert.
Dialogue Encoder Following Liu et al. (2021), the dialogue encoder is implemented with BlenderBot (Roller et al., 2021). Given an input sequence X , we concatenate all input tokens and prepend
with a [CLS] token, e.g., for the dialogue context, getting [CLS]⊕x1⊕ y1 . . .⊕xt−1. The sequence is fed into the dialogue encoder to obtain the hidden state HX . We denote the sequence representation derived from [CLS] as hX .
Emotion Experts To track possible transitions of user’s emotional states, emotion experts are associated with contextual and future user emotion predictions. We extract M fine-grained emotional reactions for each utterance in the corpus, which are inferred from COMET (Bosselut et al., 2019) using the “xReact” relation. Since emotional reactions are often emotional words (e.g., happy, sad), we use VAD (Mohammad, 2018) to identify the emotional polarity of each word according to its valence as a positive or negative emotional category. The high-frequency categories are finally retained as supervised labels for the emotion prediction task.
We divide contextual emotion experts into positive and negative emotion experts, which are two MLP transforming HX into HX,pos and HX,neg:
HX,pos = MLPpos (HX) , HX,neg = MLPneg (HX) . (1)
We project the [CLS] representations hX,pos and hX,neg of positive and negative experts to predict positive and negative emotion, respectively:
Ppos = softmax (W poshX,pos) , Pneg = softmax (W neghX,neg) , (2)
which is supervised by the positive and negative emotions collected in the e∗pos and e ∗ neg sets of the user’s last utterance in the dialogue context using cross-entropy loss:
Lctx−emopos = − 1∣∣e∗pos
∣∣ |e∗pos|∑
i=1
logPpos (e ∗ i ) ,
Lctx−emoneg = − 1∣∣e∗neg
∣∣ |e∗neg|∑
i=1
logPneg (e ∗ i ) .
(3)
Note that an utterance may be inferred to the emotions with different polarities due to cognitive differences (Westbrook et al., 2011; Zhou et al., 2022). For future emotion experts, we adopt the above method to get Lftr−emopos and L ftr−emo neg losses and train them to predict the positive and negative emotions of the user’s future utterance (i.e., next turn utterance). In this way, emotion experts can learn various emotion-level features by Lemo loss: Lemo = Lctx−emopos + L ctx−emo neg + L ftr−emo pos + L ftr−emo neg .
Keyword Experts To meet the need for dialogue coherence, keyword experts are associated with keyword predictions that act on maintaining coherence with contextual and future utterances. Here, a bidirectional emotion keyword graph G is constructed, which is also used in coherence rewards designing (a construction example is in Appendix A). We extract the salient keywords of each utterance in the corpus as vertices using a rule-based approach (Tang et al., 2019), and employ VAD to identify the emotional polarity of each keyword. The pointwise mutual information (PMI) (Church and Hanks, 1989) is adopted to construct bidirectional edges by characterizing the association between keyword pairs, where the forward edge depicts the keyword pairs extracted from the context and response, and the backward edge depicts the ones are from the future utterance and response. We further construct positive edges to describe the keywords with positive tail vertices, and negative edges are negative ones. Finally, each head vertex selects the tail vertices with the top PMI scores for building connections. The vertices of G serve as supervised labels for the keyword prediction task.
Contextual keyword experts are transformed similarly to emotion experts, and their [CLS] representations hctx−kwsX,pos and h ctx−kws X,neg can be obtained from positive and negative keyword experts Hctx−kwsX,pos and H ctx−kws X,neg , respectively. We infer the one-hop neighbors of contextual keywords from the “forward-positive” and “forward-negative” relations respectively in G to enhance the perception of the target keywords in the golden response. Specifically, we use attention (Bahdanau et al., 2015) to obtain fused embeddings ectx−kwspos and e ctx−kws neg :
ectx−kwspos = Attention(h ctx−kws X,pos ,E ctx−kws pos ), ectx−kwsneg = Attention(h ctx−kws X,neg ,E ctx−kws neg ),
(4) where Ectx−kwspos and E ctx−kws neg are positive and negative neighbor embedding matrices that share parameters with the dialogue encoder. We then concatenate ectx−kwspos and e ctx−kws neg with H ctx−kws X,pos and Hctx−kwsX,neg respectively at the token level, and use an MLP layer to fuse them to obtain keywordenhanced experts Hctx−kwsX,pos−kws and H ctx−kws X,neg−kws:
Hctx−kwsX,pos−kws[i] = MLP(H ctx−kws X,pos [i]⊕ ectx−kwspos ) Hctx−kwsX,neg−kws[i] = MLP(H ctx−kws X,neg [i]⊕ ectx−kwsneg ) (5) Further, we take the positive and negative key-
words in the golden response as supervision to optimize the Lctx−kwspos and L ctx−kws neg losses adopting cross-entropy (this process can refer to above emotion prediction task). Similarly, multi-hop reasoning on G, i.e., “forward → forward → backwardpositive” and “forward → forward → backwardnegative” (clarified in Appendix A), is performed to obtain keywords coherent with the future utterance. Taking the positive and negative keywords in future utterance as the prediction target, the keyword-enhanced future keyword experts can be optimized by Lftr−kwspos and L ftr−kws neg losses. In this way, keyword experts can learn various expression-level features by Lkws loss: Lkws = Lctx−kwspos + L ctx−kws neg + L ftr−kws pos + L ftr−kws neg .
Multi-task Training To make the experts retain the primitive semantics without hindering their respective diversity, we give them a minor constraint. Specifically, we average the representations of emotion and keyword experts to get hX,exp, and make it close to sequence representation hX by optimizing the MSE loss with a minor hyperparameter α:
Lmse = α
dh
dh∑
i=1
(hX [i]− hX,exp[i])2 , (6)
where dh is the dimension of hX . Then, we jointly train the multi-task MoE by optimizing Lexp loss:
Lexp = Lemo + Lkws + Lmse. (7)

heading: 4.2 MoE-based Reinforcement Learning
text: We use the standard reinforcement learning framework (Sutton and Barto, 2018) as the backbone.
State We concatenate the dialogue context and the extracted keywords as the initial state s1 ∈ S, i.e., s1 = {C,Ckws} (we omit the subscript t of dialogue context Ct for simplicity). At each step, the prompt token sequence E generated by the policy determined expert (i.e., action) triggers an update of the state. We record the observed state sk ∈ S at k-th step, i.e., sk = {C, E1, . . . , Ek−1}, which is encoded by the dialogue encoder to get HS,k and hS,k. We concatenate sequence representations of historical states to obtain current state embedding sk = hS,1 ⊕ . . . ⊕ hS,k. If k is smaller than the set maximum iteration steps K, we pad sk with zeros for fixing dimension. Note that when k > 1, we discard the keywords Ckws because: (1) It has already acted on the first iteration; (2) The input sequence length is limited due to the constraint of the pre-trained model (i.e., BlenderBot).
Action The action space Ak at k-th step is defined as the multi-task associated experts transformed by state sk. At state sk, our agent learns to choose an expert in Ak as expert action ak. We utilize a BlenderBot-based dialogue decoder to generate expert prompt Ek of ak. Policy Besides the above dialogue encoder as the semantic encoding policy network, we design an expert selection policy network using REINFORCE with baseline (Sutton and Barto, 2018) that includes an actor network and a value network. Actor learns an expert finding policy πφ (ak, sk,Ak) which selects the appropriate expert action ak based on the current state sk and action space Ak by emitting the probability distribution of actions in Ak. The value network measures the value Qδ (sk) of state sk as the baseline in REINFORCE. Their network structures are defined as:
ok = η ((η (skW 1)W 2)) ,
πφ (ak, sk,Ak) = ϕ (Ak ⊙ okW φ) , Qδ (sk) = okW δ,
(8)
where η(·) is an ELU activation function with a dropout layer, ⊙ is the hadamard product, ϕ(·) is the softmax function. Ak is a binarized vector for pruning the action space, and we set it as a full-one vector due to the small number of experts.
Rewards To guide policy learning, we reward the decision made at each step by measuring how well the response generated from updated state sk+1 provides ES and maintains dialogue coherence.
(1) Conversation-level ES Reward: aims to dynamically adjust the elicitation intensity of positive emotion as the conversation progresses defined as:
PEDcES = fES(y)− fES (ct) ,
rcES = T∑
t=1
cos( π 2 · t MT ) · PEDcES . (9)
Here, fES(·) measures the positive emotion level of an utterance using the emotion classification model developed by Hartmann (2022). The model is trained on six datasets containing diverse text types and achieves 66% accuracy for emotion classification. Positive emotion scores are collected as positive level. We encourage the positive emotion distance PEDcES of the generated response y and the contextual user’s post ct: (a) is non-negative, i.e., expressing empathy (equal to 0) or elicitation (greater than 0) is the underlying requirement; (b)
synchronously increases with the dialogue turn t, i.e., the early stage of the conversation is dominated by empathy, and the latter is elicitation. MT is the maximum turn of conversation, T is current turn.
(2) Turn-level ES Reward: aims to capture the feedback of user’s next turn emotion defined as:
PEDtES = |fES(y)− fES (cf )| ,
rtES = cos( π 2 · T MT ) · cos(π 2 · PEDtES).
(10)
Here, PEDtES measures the relative positive emotion distance between the generated response y and the user’s future (i.e., next turn) utterance cf . We encourage PEDtES to get smaller with the approaching of current turn T to MT , i.e., supervising smooth elicitation in the latter stage and improving tolerance to emotional fluctuations.
(3) Contextual Dialogue Coherence Reward: aims to constrain generated response y to maintain coherence with context C by measuring their coherence at keyword-level and sentence-level. First, we reconstruct a dataset (Liu et al., 2021) containing coherent and incoherent context-response pairs, where the response of the incoherent pairs is an utterance randomly sampled from the dataset. Next, a BERT-based (Devlin et al., 2019) text classification model fcDC is trained by feeding sentencekeyword pairs and achieves 85% accuracy. We take the coherence probability as the coherence score, the reward is defined as:
rcDC = fcDC (C ⊕ Ckws, y ⊕ ykws) · e Nc,kws |ykws| −1, (11) where ykws is the keyword set of y and Nc,kws is the number of keywords in ykws that are the forward neighbors of contextual keywords in G.
(4) Future Dialogue Coherence Reward: aims to introduce the consideration of coherence with the user’s future utterance cf . Similarly, we reconstruct a dataset (Liu et al., 2021) containing coherent and incoherent future utterance-response pairs and train another text classification model ffDC which achieves 77% accuracy. The reward is defined as:
rfDC = ffDC (cf ⊕ cfkws , y ⊕ ykws) · e Nf,kws |ykws| −1, (12) where Nf,kws is the number of keywords in ykws that have a backward relation with keywords cfkws of cf in G.
(5) Total reward. The total reward is r = wcES ∗ rcES+wtES ∗rtES+wcDC ∗rcDC+wfDC ∗rfDC .

heading: 4.3 Optimization
text: We set K-step iterations, and the goal of agent learning is to maximize the expected cumulative reward: Jθ = Eπ [∑K k=1 γ krk+1 ] , where θ is the learned parameter and γ is the discount coefficient. The agent is optimized by Lagent loss and its policy gradient is defined as:
∇θJθ = Eπ[∇θ log πφ(ak, sk,Ak)(G−Qδ(sk))], (13) where G is the discounted cumulative reward from the initial state to the terminal state. Finally, we take the hidden state HS,K+1 of the state sK+1 to generate the response, where the decoder is optimized by Lgen loss:
Lgen = − M∑
m=1
logP (ym | HS,K+1, y<m). (14)
Warm Start We use the pretrained small version of BenderBot for initializing our model. The initial state is used as input to fine-tune the model for warm start by optimizing Lwarm = Lexp + Lgen.
Joint Training Our model is finally jointly trained by optimizing Ljoint loss:
Ljoint = Lagent+Lgen+ 1
K + 1
K+1∑
k=1
Lexp,k (15)

heading: 5 Experiments

heading: 5.1 Experimental Setup
text: Dataset Our experiments are conducted on the widely used ESConv (Liu et al., 2021), a multi-turn conversation dataset for ES. In a conversation, the user confides personal negative situation, and the supporter provides comfort and support to improve the user’s mental state. The statistics of ESConv and graph G after preprocessing are in Table 1.
Baselines (1) MoEL (Lin et al., 2019): An empathetic conversation model that uses multiple decoders to capture possible user emotions for generating. (2) MIME (Majumder et al., 2020): An empathetic conversation model that mimics user’s emotions during responding. (3) BlenderBot-Joint (Liu et al., 2021): An ESC model that prepends a predicted strategy token on the backbone of BlenderBot. (4) MISC (Tu et al., 2022): An ESC model that fuses commonsense. (5) GLHG (Peng et al., 2022): A commonsense-based ESC model that designs a global-to-local graph. (6) We design Bart-Joint by replacing the backbone of BlenderBot-Joint with Bart (Lewis et al., 2020). It achieves comparable performance to MultiESC (Cheng et al., 2022) as its replacement since MultiESC’s code is unavailable.
Implementation Details We implement all models with Pytorch, and all pretrained models (i.e., BlenderBot, Bart) use small versions. We set the number of steps K = 2 and reward weights wcES = wcDC = 0.1, wtES = wfDC = 1.0 (selected using a grid-search approach with two values {0.1, 1.0} for each hyperparameter). We extract M = 10 emotional reactions for each utterance. The maximum number of conversation turn MT is set to 10. The discount factor γ is 0.99, the hyperparameter α is 1e-5, and the batch size is 16. We use Adam optimizer (Kingma and Ba, 2015) with an initial learning rate of 2e-5 and a linear warmup of 120 steps for training on a GPU-V100 machine. The warm start stage is trained for 5 epochs, and the joint training stage is set to 3 epochs. The decoding settings are consistent with Liu et al. (2021). For a fair comparison, all baselines with available codes are reproduced under the same setting.

heading: 5.2 Automatic Evaluation
text: We adopt Perplexity (PPL), Bleu (B-n) and Distinct (D-n) to evaluate the general generation quality and diversity of the models. To measure how well the generated responses achieve goals, we define (1) ES scores containing conversation-level (cES) and turn-level (tES), i.e., rcES and rtES , measure the elicitation intensity of positive emotion involving conversation progress and the perceived intensity to the user’s next turn emotion; (2) Dialogue coherence scores containing contextual (cDC) and future (fDC), i.e., rcDC and rfDC , measure the coherence with the context and the user’s future utterance.
Overall Performance In Table 2, compared with all baselines, our SUPPORTER achieves the most diverse expressions and highest ES (12.9% outperforms the second best MoEL on cES) while maintaining competitive dialogue quality (PPL, Bleu) and coherence (cDC, fDC). Supportive responses generated by MoEL are often accompanied by low diversity and low coherence due to the retelling of generic responses (e.g., “I am glad I could help you” with high positive emotion) that are found from its outputs. Bart-based models benefit from robust sequence modeling (Lewis et al., 2020) with inherent advantages in coherence and Bleu but perform poorly in ES and diversity. The contextual coherence (cDC) of our SUPPORTER is inferior to BlenderBot-Joint, which is acceptable as ES for positive emotion elicitation needs to sacrifice a little coherence to jump out of negative topics.
Ablation Study In Table 2: First, we remove the emotion experts (w/o EmoExperts), keyword experts (w/o KwsExperts), and the multi-task as-
sociated with the experts (w/o Multi-Task), respectively. Emotion experts mainly act on ES, including cES and tES. Keyword experts contribute significantly to dialogue coherence, including cDC and fDC. Multi-task training endows experts with specific abilities and thus has an impressive impact on overall performance. Second, we remove the ES rewards (w/o ESRewards) and dialogue coherence rewards (w/o DCRewards), respectively. The former improves positive support, and the latter maintains grounded expression. Therefore, besides achieving their own goals, they also benefit dialogue diversity and quality, respectively. Moreover, we replace the expert selection policy network with random sampling (w/o ExpertPolicy). Random experts lead to uncertainty in decision-making and thus damage overall performance, especially on ES and coherence. Third, we test using only warm start and without joint training (Warm-Start Only) as well as without warm start and only joint training (w/o Warm-Start). The former reaches comparable or even worse results than the baselines, and the latter greedily achieves the goal of maximizing the rewards resulting in low dialogue quality.

heading: 5.3 Interactive Human Evaluation
text: We recruited three crowdsourcing workers and exposed them to 100 negative situations randomly sampled from the test set. They were asked to engage in multi-turn conversation with the models to simulate the process of seeking ES and to choose the better one (Win) from a model pair by considering five aspects, respectively: (1) Fluency: which bot’s response is more fluent and understandable? (2) Informativeness: which bot’s response is more diverse and specific, and contains more information? (3) Coherence: which bot’s response is more coherent with context in a multi-turn conversation? (4) Supportiveness: which bot provides more effective ES, i.e., is more likely to elicit users to change their emotions from negative to positive? (5) Overall: generally, which bot is more preferred?
As in Table 3, from the comparison with baselines, we found that a single incoherent response (cDC in Table 2) has less impact on the coherence of the overall multi-turn conversation. Comparisons with variants of SUPPORTER demonstrate that key components of our model, i.e., emotion experts and expert selection policy, lead to significant advantages in the overall performance.

heading: 5.4 Qualitative Analysis
text: Specificity of Experts To analyze the quality of the experts, we show the specificity of the experts learned by SUPPORTER. As shown in Figure 3, we visualize the latent space of experts using t-SNE on 200 conversation samples. The latent space distributions of multi-task-associated experts are clearly separated and clustered in specific regions. Some overlap is also intuitive due to the similarity between experts with the same polarity, e.g., contextual and future positive emotion experts. This verifies our MoE has diverse and specific semantics and the superiority of multi-task learning.
Adjustability of Elicitation To further explore the adjustability of elicitation intensity of positive emotion in multi-turn conversation, we analyze the trend of positive emotion distance with the dialogue
turns, i.e., PED = fES(y) − 1T ∑T
t=1 fES (ct). As shown in Figure 4, the PED score of all models tends to rise first and then fall. In the early stage of the conversation (turn<6), SUPPORTER keeps the same trend as the empathy model (i.e., MoEL, MIME) and gradually increases the intensity of elicitation. This is attributed to our encouragement that it should progressively transform the conversation from empathy-dominated to elicitation-dominated. In the later stage of the conversation (turn>6), SUPPORTER still maintains a higher level of elicitation than baselines and shows robust adjustment ability.

heading: 5.5 Parameter Analysis
text: We further analyze the impact of the number of iteration steps K. In Table 4, with the increase of steps, diversity and tES show an upward trend, while other metrics show a downward one. This happens possibly because the informativeness of the generated responses increases with selected experts, making it possible to lose focus and thus lead to poor dialogue quality. Furthermore, SUPPORTER outperforms the best baselines in most cases, confirming its effectiveness.

heading: 6 Conclusions
text: In this paper, we introduce a new paradigm to formalize multi-turn ESC as a process of positive emotion elicitation and propose an MoE-based rein-
forcement learning model SUPPORTER with welldesigned ES and dialogue coherence rewards. Extensive experiments verify the superiority of our model in providing effective ES for positive emotion elicitation while maintaining conversational goals including coherence. Our work will facilitate future work to develop ESC with positive emotion elicitation for improving the users’ mental state.
Limitations
We discuss three limitations of this work as follows. The first one is the instability of reinforcement learning. Reward-driven policy learning is an essential advantage of this work because it is better equipped with the positive emotion-driven process of ESC than existing works and can model flexible ESC expression beyond the training data. However, this flexibility also suffers from instability, which calls for additional knowledge or strategies to refine the learning process.
The second one is the need for further reference to psychological theory. An advantage of our work is to learn posterior ESC patterns integrating the dialogue context and future feedback in the form of rewards. However, there is still other valuable prior knowledge to be referred from psychology studies, e.g., the CBT (cognitive-behavioral therapy) methods. This kind of prior knowledge can be used as additional knowledge to refine the learning process as mentioned in the first limitation.
The third one is that the reward design can be further optimized. The ideal case is to construct a high-quality dataset with human-feedback labels for training reward model (e.g., the constructed example of ChatGPT). At the same time, the larger parameter of the reward model, the more conducive it is to learn a robust policy and avoid it overfitting to the reward function. However, such optimizations need a trade-off with cost.
Ethical Considerations
In this paper, the ESConv dataset used in our experiments is a publicly-available benchmark for emotional support conversation, which does not contain sensitive and personal information as well as unethical language. Our work builds on this dataset to study positive emotion elicitation to improve the user’s mental state. Therefore, we focus on constructing a dialogue system to provide emotional support from families and friends in the daily scenarios limited by this dataset rather than profes-
sional psychological counseling or psychological treatment. For risky non-daily scenarios such as self-harm or suicide-related conversations, we do not claim that the dialogue system we built has a treatment or improvement effect on them. Additionally, we also ensure the anonymity of our interactive human evaluation. We believe our work meets ACL’s Code of Ethics.

heading: Acknowledgements
text: This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2020GQG0005. This work was also supported by Tsinghua Precision Medicine Foundation.
This work was also supported by the National Natural Science Foundation of China (with No. 62272340, 61876128, 62276187).

heading: A Bidirectional Emotion Keyword Graph
text: A construction example of the bidirectional emotion keyword graph G is in Figure 5.
One-hop Reasoning on Graph G For the contextual keyword “close”, its one-hop neighbor reasoned by the “forward-positive” relation is “understand”, and the one reasoned by the “forwardnegative” relation is “frustrated”. Further, the one-hop neighbors reasoned by the “forward” relation are the union of the one-hop neighbors of the above two relations, i.e., “understand” and “frustrated”. For the keyword “frustrated” of the response, it cannot reason the one-hop neighbor using the “backward-positive” relation. Therefore, its one-hop neighbors reasoned by the “backward” relation are the same as the one-hop neighbors reasoned by the “backward-negative” relation, i.e., “close”, “warning”, and “pandemic”.
Multi-hop Reasoning on Graph G Taking the “forward → forward → backward-positive” multihop reasoning as an example, using the “forward” relationship for the contextual keywords to perform one-hop reasoning can obtain the set of neighbors that contain the keywords of the response, which we regard as the extended keyword set of the response determined by the context. Using the keywords in this set as a starting point to perform the second-hop reasoning by the “forward” relation can result in the expanded keyword set of the user’s future utterance (i.e., the user’s next turn utterance) determined by the response. Further, similarly, the third-hop reasoning is performed using the “backward-positive” relation to determine the extended positive keywords set of the response coherent to the future utterance.

heading: B Case Study
text: In Table 5, two cases in multi-turn conversation generated by five models are selected. We found that the advantage of SUPPORTER is that it is more likely to elicit users’ emotions to transit positively by adaptively adjusting the intensity of empathy and elicitation. For example, low turns have a tone of strong empathy (e.g., “that is so awful”, “you are struggling financially and that you have not been able to afford your kids presents”). As the conversation deepens, the elicitation intensity of positive emotion progressively increases (e.g., “do you know anyone who has gone through similar situations?”, “it is okay to be proud of yourself for being able to make them”), and finally the expression with strong elicitation will help users jump out of negative situations (e.g., “I am sure you will be successful in the new job market”, “you can really enjoy seeing them happy”). During these multi-turn conversations, SUPPORTER effectively provides ES with diverse expressions while maintaining the dialogue coherence, being consistent with the observations from automatic and interactive human evaluations.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Sec. Limitations
3 A2. Did you discuss any potential risks of your work? Sec. Ethical Considerations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Sec. Abstract and Sec. 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.
3 B1. Did you cite the creators of artifacts you used? Sec. 4, Sec. 5
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sec. 4, Sec. 5
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Sec. Ethical Considerations
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Sec. 4, Sec. 5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Sec. 5
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Sec. Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Sec. Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sec. 5
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Sec. 5
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Limited by the space. Crowdsourcing workers are from Amazon Mechanical Turk.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Using the keywords in this set as a starting point to perform the second-hop reasoning by the “forward” relation can result in the expanded keyword set of the user’s future utterance (i. e. , the user’s next turn utterance) determined by the response. Further, similarly, the third-hop reasoning is performed using the “backward-positive” relation to determine the extended positive keywords set of the response coherent to the future utterance