Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents
Authors: Yang Deng
Wenxuan Zhang
Wai Lam
See-Kiong Ng
Tat-Seng Chua
Venue: ICLR 2024 poster (2024)
Keywords: Dialogue Policy Planning
Proactive Dialogue
Large Language Model
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Proactive dialogues serve as a practical yet challenging dialogue problem in the era of large language models (LLMs), where the dialogue policy planning is the key to improving the proactivity of LLMs. Most existing studies enable the dialogue policy planning of LLMs using various prompting schemes or iteratively enhance this capability in handling the given case with verbal AI feedback. However, these approaches are either bounded by the policy planning capability of the frozen LLMs or hard to be transferred to new cases. In this work, we introduce a new dialogue policy planning paradigm to strategize LLMs for proactive dialogue problems with a tunable language model plug-in as a plug-and-play dialogue policy planner, named PPDPP. Specifically, we develop a novel training framework to facilitate supervised fine-tuning over available human-annotated data as well as reinforcement learning from goal-oriented AI feedback with dynamic interaction data collected by the LLM-based self-play simulation. In this manner, the LLM-powered dialogue agent can not only be generalized to different cases after the training, but also be applicable to different applications by just substituting the learned plug-in. In addition, we propose to evaluate the policy planning capability of dialogue systems under the interactive setting. Experimental results demonstrate that PPDPP consistently and substantially outperforms existing approaches on three different proactive dialogue applications, including negotiation, emotional support, and tutoring dialogues.

Published as a conference paper at ICLR 2024
PLUG-AND-PLAY POLICY PLANNER FOR LARGE LAN- GUAGE MODEL POWERED DIALOGUE AGENTS
Yang Deng' Wenxuan Zhang' Wai Lam? See-Kiong Ng¹ Tat-Seng Chua National University of Singapore The Chinese University of Hong Kong {ydeng, seekiong, dcscts /onus edu.s9 isakzhang@gmail. com wlam@se. cuhk. edu. hk
ABSTRACT
Proactive dialogues serve practical yet challenging dialogue problem in the era of large language models (LLLMs), where the dialogue policy planning is the key to improving the proactivity of LLMs. Most existing studies enable the di- alogue policy planning of LLMs using various prompting schemes or iteratively enhance this capability in handling the given case with verbal AI feedback. How- ever, these approaches are either bounded by the policy planning capability of the frozen LLMs hard to be transferred to new cases. In this work, introduce new dialogue policy planning paradigm to strategize LLMs for proactive dialogue problems with a tunable language model plug-in plug-and-play dialogue pol- icy planner, named PPDPP. Specifically, we develop a novel training framework facilitate supervised fine-tuning over available as reinforcement learning from goal-oriented Al feedback with dynamic interac- tion data collected by the LLM-based self-play simulation. In this manner, the LLW-powered dialogue agent can not only be generalized to different cases after the training, but also be applicable to different applications by just substituting the learned plug-in. In addition, we propose to evaluate the policy planning ca- pability of dialogue systems under the interactive setting. Experimental results demonstrate that PPDPP consistently and substantially outperforms existing ap- proaches on three different proactive dialogue applications, including negotiation, emotional support, and tutoring dialogues.
data as well
INTRODUCTION
Large language models (LLMs) powered dialogue agents (e.g., ChatGPT (Ouyang et al., 2022), Vi- cuna (Chiang al., 2023), LLaMA2-Che. (Touvron et al., 2023), etc) have demonstrated exceptional proficiency in context understanding and response generation in various dialogue problems (Bang al., 2023; Zhang et al., 2023b; Zhao et al., 2023). However, as LLMs are trained to passively fol- low users' instructions, dialogue agents built upon them typically prioritize accommodating users' intention. Therefore, LLM- -oweee dialogue agents often face challenges handling proactive dialogue problems that require the dialogue agent to strategically take the initiative to steer the con- versation towards an anticipated goal (Deng et al., 2023a), such as negotiation (Zhan et al., 2022), emotional support (Liu et al., 2021; Zheng et al., 2023), and tutoring (Macina et al., 2023).
In such scenarios, the key is to improve the capability of -oweeed dialogue agents in dialogue policy planning, which refers to the process of deciding what actions the dialogue agent should take to effectively achieve specific goals during the dynamic interactions with the user. In the pre-LLM era, researchers mainly employ corpus-based learning approaches conduct the dialogue policy planning via dialogue act prediction (Zhou et al., 2020; Joshi et al., 2021; Cheng et al., 2022; Wang al., 2023c; Deng et al., 2023c; 2022). However, such approaches rely heavily on static human- annotated dialogues and fail to optimize the long-term goal of the conversation. With the advent of LLMs, it further becomes unrealistic and costly to fine-tune the whole dialogue systems for every specific application. To this end, as shown in Figure 1(a), recent works investigate prompt-based
"Corresponding author.
'The code can be accessed via https: / //gittuut www.tttt.....enneeeeeeeeeee com/deggyagggl 7/ 7/PPPP..

---

Published as a conference paper at ICLR 2024
policy planning methods that prompt frozen actor LLM to either conduct self-thinking of strategy planning for each turn (Zhang et al., 2023a; Deng et al., 2023b; Wang et al., 2023a) or generate AI feedback given the whole dialogue history to iteratively improve the dialogue policy planning for certain case (Fu et al., 2023; Yu et al., 2023).
Despite their effectiveness in improving the dialogue policy planning, there are several challenges that remain to be tackled. 1) LLMs fall short of planning effective dialogue policy with zero-shot or few-shot prompting schemes (Deng et al., 2023b). Therefore, the improvement of accomplishing the goal will be limited by the planning capability of the frozen actor LLM. 2) Existing approaches based on iterative refinement (Fu al., 2023; Yu et al., 2023) lack of transferabilityy as multiple rounds of self-play dialogue simulations are required for every new-coming case to plan satisfactory strategy for it, which impractical in real world applications. 3) Existing studies typically evaluate the performance of dialogue agents in terms of turn-level response quality measurements based on fixed reference responses. However, these evaluation protocols fail to automatically assess the policy planning capability of the dialogue agent, which is determined by the effectiveness and efficiency of the goal achievement in multi-turn conversations.
To tackle these challenges, we introduce novel di- alogue policy planning paradigm to strategize LLMs with a tunable language model plug-in, named Plug- and-Play Dialogue Policy Planner (PPDPP). As shown in Figure 1(b), PPDPP acts the policy agent to pre- dict the dialogue strategy at the next turn for the dia- logue agent, which can be first supervisedly fine-tuned with available human-annotated corpora. Then, we employ the self-play paradigm to prompt two LLMs (an assistant and a user) with various case background information to perform the role-playing conversation that simulates the dynamic environment of multi-turn interactions between the dialogue agent and the real user. For each case, these two LLMs are each tasked with distinct, often competing goals (e.g., in negoti- ation dialogues, the buyer seeks attain a more fa- vorable price, whereas the seller endeavors to secure higher price). Meanwhile, third LLM acts the reward model to provide goal-oriented verbal feed- back, indicating the goal achievement, which is trans- formed to scalar rewards used for reinforcement learn- ing (RL). When reaching the goal or the maximum conversation turn, we leverage RL algorithm to fur- ther tune the policy agent with the collected interac- tion data and the goal-oriented AI feedback. In this way, the LLM-pooered dialogue agent can not only cases than
Case
System Utterance
User L.M
Info.
Dialog LLM
User Utterance
Action
Actor LLM
Dialogue
Prompt
History
(a) Prompt- Policy Planning
Case Info.
System Utterance
User LLM
Dialog LLM
User Utterance
Action Prompt
Dialogue History
Interenve
Reward LLM
Data
PPDPP
Trained
Trained with RL
with SFT
exhibit more adaptability various prompt-based approaches. but also find utility across diverse applications simply by shifting the tuned plug- in without affecting the LLM's exceptional capabilities of context understanding and response generation.
(b) Plug-and-Play Dialogue Policy Planner
Figure 1: of LLM-based proactive dialogue systems. Dashed lines will be blocked during the in- ference phase.
The architectures of two types
To overcome the limitation of traditional turn-level response evaluation metrics, we further propose LLM-based interactive evaluation approach that har- nesses LLM-based user simulators and reward models as introduced above. This approach enables the simulation of diverse user-assistant interactions to assess both the success rate and the average number of turns of achieving designated goals. We conduct extensive experiments on three differ- ent proactive dialogue problems, including negotiation, emotional support, and tutoring dialogues. Experimental results demonstrate the superiority of the proposed PPDPP framework over existing LLM-based dialogue systems, showing that PPDPP can effectively and efficiently lead the conver- sations to achieve the designated goal.
2

---

Published as a conference paper at ICLR 2024
Table 1: Overview of L.LM general (upper) and dialogue (lower) policy planning methods.
Subjective Goal
Multi-turn
Decision Explicit Making
Long-term Optimization Learning
Gradient
Transfer- ability
Interaction
Strategy
DSP (Li et al., 2023)
RAP (Hao et al., 2023)
X
Reflexion (Shinn et al., 2023)
x
Retroformer (Yao et al., 2023)
1
MI-Prompt (Chen et al., 2023)
X
1
X
Ask-an-Expert (Zhang et al., 2023a)
ProCoT (Deng et al., 20233)
X
ICI-AIF (Fu et al., 2023)
X
PPDPP
/
V
V
2
RELATED WORKS
Dialogue Policy Planning Dialogue policy planning has been widely-studied in task-oriented di- alogues (Jang et al., 2022; Feng al., 2023) and conversational recommendation (Gao et al., 2021; Deng et al., 2021), where the interaction process can be easily abstracted into a sequence of slots and values (e.g., location, price, etc). Meanwhile, the success of planning is objective, such as whether the system provides an appropriate entity/item. However, in proactive dialogues (Deng et al., 2023a; Liao et al., 2023), there is no pre-defined agenda or schema for simplifying the multi-turn interac- tion. Instead, the natural language interaction requires more complex reasoning and certain domain knowledge (e.g., psychological or pedagogical skills). Moreover, the planning outcome is rather subjective, such as learning gain during tutoring, emotional intensity relaxation during counselling. Therefore, it imposes more difficulties in planning optimal dialogue policy in proactive dialogues. In order mimic the behaviors of human experts, corpus-base -bspud fine-tuning approaches are typically adopted for predicting the dialogue strategies (Joshi et al., 2021; Cheng et al., 2022; Wang et al., 2023c). As summarized in Table I, we differentiate our method from the recent LLM-based policy planning methods in terms of seven perspectives. General policy planning methods typically opti- mize towards an objective goal in a single-turn interaction, such as ROUGE score in summarization (Li et al., 2023) or accuracy in QA (Shinn et al., 2023; Yao et al., 2023). As for dialogue policy planning methods, Chen et al. (2023) validate the effectiveness of mixed-initiativv strategy- prompting in proactive dialogue problems. Some methods Wang et al. (2023b); Deng al. (2023b); Zhang et al. (2023a) prompt LLMs to conduct self-thinking of policy planning for the next turn, ig- noring the long-term conversation goals. Fu et al. (2023) conduct self-play simulation iteratively refine the policy planning with long-term feedback. However, this type of iterative refinement ex- clusive each individual case, but not transferable new situations. Moreover, the policy planning capability in LLM-powered dialogue agents cannot be improved by these methods, as all parameters are frozen and not learnable. As for the proposed PPDPP, learnable language model plug-in in can be fine-tuned for improving the policy planning capability without affecting other functionalities of L..M-poeered dialogue agents.
Learnable Plug-ins for Large Language Models Due the black-box nature of commercial LLMs and the high expenses of fine fine-tuning the whole open-source LLMs, a recent trend in improv- ing certain capabilities of LLMs is to investigate the utility of external plug-ins, such as APIs (Schick et al., 2023), vision models (Wu et al., 2023), or functional models from Huggingface (Shen et al., 2023). However, these plug-ins fail learn from valuable feedback to iteratively enhance the their capabilities, resulting in performances that are solely dependent Some latest studies demonstrate that small language models are also valuable plug-ins for LLMs in various applications, such text classification (Xu et al., 2023), summarization (Li et al., 2023), question answering (Yao et al., 2023), or to endow LLMs with specific capabilities, such as mental state reasoning (Sclar et al., 2023). These learnable plug-in -nn can benefit from supervised fine-tuning with available resources or reinforcement learning with the environment.
the quality of fixed plug-ins.
Reinforcement Learning from AI Feedback As LLMs become more powerful to be capable of supervising other models, Bai et al. (2022) propose the idea of "RL from AI Feedback" (RLAIF) to train a harmless and detoxified LLM through self-impprvement, without human labels. Since the feedback from LLMss is typically in the form of natural language, most of existing studies (Shinn
3

---

Published as a conference paper at ICLR 2024
et al., 2023; Fu et al., 2023; Madaan et al.. 2023; Hao et al., 2023) directly leverage the generated natural language feedback from LLMs to self-refine the task instruction prompt, instead of obtaining scalar reward for training the model. In this work, we propose the goal-oriented AI feedback for facilitating RLAIF under the context of dialogue systems, which not only transforms the textual feedback into scalar rewards, but also capture long-term goal-oriented rewards that obtain from the dynamic multi-turn interactions, instead of AI preference on single-turn responses.
3 METHOD
MDP Environment We formulate the dialogue process as Markov Decision Process (MDP). At each turns i, according to the observation on the dialogue history, the dialogue system selects an action at E A. where is a set of candidate strategies pre-defined by domain experts. return, the user player responds to the action. This process repeats until the conversational goal is achieved or the maximum number of turns T reached. The objective is to learn a policy æ maximizing the expected cumulative rewards over the observed dialogue episodes as:
Σ r(st, a.)
n arg max EII
(1)
t=U
where 8y is the state representing the dialogue history. r(-) is the intermediate reward, denoted as ry-
Plug-and-Play Dialogue Policy Planner As shown in Figure 1(b), smaller model is adopted to act as a plug-in for controlling the dialogue policy planning in the LLM-powered dialogue agent. We leverage tunable pre-trained language model, e.g. RoBERTa (Liu al., 2019), the dialogue policy planner to predict the action a. Before performing interactive online learning, PPDPP can be initialized by supervised fine-tuning (SFT) on available dialogue corpus D. Specifically, given the dialogue history (u]". u ..... " u } as the current state 8t, the SFT process aims to minimize the cross entropy loss between the predicted action 0r and the human-labeled action y/. for each turn t in the annotated dialogue:
= PPDPP U1
SyS
USE
SyS
USF
(2)
.....
C. DD] dED Ta
Σ
1 Ta
at
log yt
(3)
r=1
where Ta denotes the number of turns of the dialogue. Although corpus-based learning typically leads to sub-optimal policy, such initialization is supposed to accelerate the convergence process of interactive online training.
Self-play Interaction During the interactive online learning, we prompt two LLMs as the user and the assistant to perform self-play conversations that simulate the dynamic user-assistant inter- action. The descriptions of the role and the instructions of their corresponding conversational goals are delivered to each LLM. For example, in emotional support dialogues, the patient (user) player will receive the situation description about the causes of the emotional problem while the thera- pist (assistant) player will receive the task description reduce users' emotional distress and help them work through the challenges. In tutoring dialogues, the student (user) player will receive the descriptions about their knowledge state, while the teacher (assistant) player will receive the task de- scriptions to teach users to master a certain exercise. When it comes to the assistant's turn, PPDPP first predict the next action a, based on the interaction history. The predicted action mapped to a pre-defined natural language instruction (a)).). Then the assistant player generates the strategic response based on the dialogue history and the natural language action instruction:
LLMsys Ma(a)); ui"s ud u 11-1, 1
sys
(4)
Then the user player generates the response based on the updated dialogue history with un n,
2 usr
LLM (Puss: sys usr u1 ..... u4 u111 (
sys
(5)
where Psys and Pusr are the corresponding prompts. This process is repeated until a terminal state is reached. Overall, there are three types of states in the self-play interaction: 1) ON-GOING: the dialogue between the two players is still ongoing as the goal has not been achieved; 2) GOAL- COMPLETED: the designated conversational goal is completed, such as solving the seeker's emo- tional problem or the student mastering the exercise; 3) GOAL-FAILED: the conversational goal is considered failure when the conversation reaches maximum turn without completing the goal.
4

---

Published as a conference paper at ICLR 2024
LLM as Reward Model We prompt third LLM to be the reward model, named LLM which has two functions: (1) to determine the goal completion during the conversation; (2) to evaluate the policy outcome with scalar rewards. Specifically, we prompt the reward model to answer multi- choice question to generate the goal-oriented Al feedback. We further define a mapping M, (-) to transform verbal feedback to scalar rewards.
Due to the subjectivity of the planning outcome as well as the variance of the LLM- --M-eeeratted output, follow common practice (Wang et al., 2023e) to alleviate these issues by sampling the decoded sequences of the reward LLM. In general, we obtain a scalar value Vt by sampling the goal-oriented Al feedback for 1 times and converting them into a scalar value through averaging:
et - Σ (LLLLLModPPPod uI"
usr
U U2
UsE
Eye
2er (T)):
(6)
where Prwd is the prompt. We first use V; determine the state of the self-play interaction. If ot not less than a certain threshold e, we regard the state as GOAL-COMPLETED. If the conversation reach a terminal state, including COAL-CCMPLETED and GALLEF we have the reward r; et- If not, we assign small negative reward, e.g., -0.1, to penalize the lengthy conversation for promoting efficient goal completion.
Reinforcement Learning When reaching the goal the maximum conversation turn, obtain goal-oriented reward rj. We denote the policy agent as π {a,)s)), which returns the probability of taking action ae given the state 8t. To optimize the policy agent, we simply use the vanilla policy gradient method (Sutton et al., 1999), formulated as follows:
+ 0 V log No (ar]ss))
where 0 denotes the parameter of the policy network, a denotes the learning rate of the policy net- work, and R, is the total reward accumulating from turn f to the final turn T: R₂ =I where y is discount factor which discounts future rewards over immediate reward. = rv,
T
During the inference, the tuned PPDPP directly provides the action prompt, based on the dialogue history, for guiding the dialogue L.LM to generate the next response, while the reward L.MM will not be used as shown in Figure 1(b). In this manner, the LLM-powered dialogue agent with the tuned PPDPP can be directly applied to diverse new situations without the necessity of performing multiple iterations of simulation for every cases.
4 EXPERIMENTAL SETUPS
4.1 DATASETS
We evaluate the proposed framework three different applications of proactive dialogues, including negotiation dialogues, emotional support dialogues, and tutoring dialogues. The statistics of adopted datasets are presented in Table 2. In specific, the human-annotated dialogues in the train set are used for the supervised fine-tuning of the dialogue policy planner, while only the case background information in the dataset is adopted for the reinforcement learning process.
CraisglistBargain (He et al., 2018) created under the bargain negotiation setting where the buyer and the seller are negotiating the price of an item on sale. Yang et al. (2021) design 15 di- alogue acts for labeling this dataset, including 11 negotiation strategies and 4 terminal acts. In our experiment, we only consider the 11 nego- tiation strategies and split the development set into two parts as a new development set and a test set. Each case is associated with an item cat- egory, an item description, buyer target price, and seller target price, which are adopted as the instruction information.
Table 2: The statistics of datasets (train/dev/test).
Dataset
# Case
# Act
CraisglistBargain
3,090/1 88/11888188
ESConv CIMA
1,040/130/13
8
909/1
ESConv (Liu et al., 2021) is emotional support conversation dataset, contains 1,300 cases with 8 types of support strategies. We adopt the original train/dev/test split. Each case is accompanied with problem type, an emotion type, and situation description.
5

---

Published as a conference paper at ICLR 2024
CIMA (Stasaski et al., 2020) is crowd-sourced dataset, where annotators were asked to role-play students and teachers by working through an exercise with pedagogical strategies on translating a prepositional sentence from English to Italian. We regard each exercise case in our experiment and randomly split the dataset into train/dev/test sets by 8:1:1.
4.2 EVALUATION PROTOCOLS
Previous studies (Joshi et al., 2021; Cheng et al., 2022; Wang et al., 2023c) on emotional sup- port, negotiation, and tutoring dialogues typically evaluate the turn-level performance, based on the fixed reference responses. Differently, when it comes to the evaluation of dialogue pol- icy planning, it would be more appropriate focus on the dialogue-level interactive evaluation (Kwan et al., 2023; Gao et al., 2021). Goal completion is the key to evaluating proactive dia- logue systems (Deng et al., 2023a). To this end, we adopt the average turn (AT) and the suc- cess rate at turn / (SR@1) () as the automatic evaluation metrics. AT measures the efficiency of the goal completion by calculating the average number of turns to achieve the goal while SR measures the effectiveness of the goal completion by computing the success rate of achieving the goal within a pre-defined maximum turn. We set the maximum turn of the conversation as 8 in our experiment. In particular, as for Craisglistaargain, since reaching a deal cannot be regarded as a success, we adopt the Sale-to-List Ratio (SL%) (Zhou et al., 2019) as the met- rics to determine the effectiveness of goal completion during the negotiation dialogue, which is formulated as (deal price seller target price)/(buyer target price seller target price). A higher SL% represents the buyer gets more benefits from the deal. If failing to reach a deal at the end, we assign SL% as 0 for this case. The evaluation is con- ducted by interacting with the LLM-based user simulator, while the goal completion (i.e., GOAL- COMPLETED or GOAL-FAILEDD is determined by the I.L M-based reward model, as introduced in Section 3. To validate the reliability of this evaluation framework, a preliminary analysis of adopting LLMs as reward models and user simulators is presented in Appendix A.
4.3 IMPLEMENTATION DETAILS AND BASELINES
We adopt RoBERTa (roberta large) as the default plug-and-play dialogue policy planner for evaluation. The details of training process are provided in Appendix B. the main results of evaluating the dialogue policy planning methods, as the frozen LLM for both the role-playing LLMs (LLM,ys and LLM and the reward model (LLMww)). As for the role-pla role-playing LLMs, set the temperature tic outputs with the same inputs. While we set 7 = 1.1 and the sample times / -= 10 for the reward model to integrate the scalar rewards2. In order to compare with different LLMs, we also adopt two popular open-source LLM-based dialogue systems, including Vicuna- - 13B 33--de (os-delta-va 1 and LLaMA-2- 1121-11----- Chat with the same ppper-parameter setting ChatGPT.
ChatGPT 5ott-3.5-turbo-055
to generate the determinis-
The role-playing prompts for the assistant and the user players (Pays and Pust) are presented in Ap- pendix E. and E.2, respectively. The mapping of natural language instructions, MMa.),, for dialogue actions is presented in Appendix E.3. The prompts for the reward model, Prwd. and the whole reward mapping process are presented in Appendix E.4.
As for the baselines, we first compare to general fine-tuned dialogue models, DialoGPT (Zhang et al., 2020). Furthermore, we adopt several latest LLM- dialogue policy planning methods for comparisons, including vanilla LLM (Standard), Proactive (Deng et al., 2023b), ProCoT (Deng et al., 2023b), Ask-an-Expert (AnE) (Zhang et al., 2023a), and ICL-AIF (Fu et al., 2023). Details for these baselines are presented in Appendix E.5.
EXPERIMENTS
5.1 OVERALL EVALUATION
Table 3 summarizes the experimental results of method comparisons three datasets. Overall, the proposed method, PPDPP, consistently outperforms all the baselines with a noticeable margin across
We conduct the ablation study on the sampling strategy in Appendix D.

---

Published as a conference paper at ICLR 2024
Table 3: Experimental results. #Tokens denotes the approximate tokens required for prompting L.MSs to simulate a new conversation, where I. denotes the standard prompt length for one episode of conversation. M and N are respectively set to 3 and 5 by default as set by corresponding works.
CraisglistBaraain
ESConv
CIMA
Method
#Tokens
A
SR1
SL
AT
SRT
AT
SR
DialoGPT
6.73
0.3245
0.2012
5.31
0.7538
5.43
0.4956
Standard
OLL)
6.47
0.3830
0.1588
5.10
0.7692
3.89
0.6903
AnE (Zhang et al., 2023a)
O[(M + 1)L)
5.91
0,4521
0.2608
4.76
0,8000
3.86
0.6549
Proactive (Deng et al., 2023b)
OLL)
5.80
0.5638
0.2489
5.08
0.7538
4.84
0,5310
+ MIIPrompt (Chen et al., 2023)
((2()
5.74
0.5691
0,2680
4.78
0.7846
4.70
0,5664
ProCoT (Deng et al., 2023b)
O(L)
6.22
0,5319
0.2486
4.75
0.7923
4.58
0,5487
+ MII-Prompt (Chen et al., 2023)
O(2L)
6.12
0,5532
0.3059
4,83
0.7769
4.72
0.5221
ICL-AIF (Fu et al., 2023)
O((N + +1)L)
6.53
0.3617
0,1881
4.69
0.8079
4.19
0.6106
PPDPP
O(L)
5.62
0.6117
0.3376
4.56
0,8462
3.03
0,8407
w/o SFT
O(L)
5.71
0.6223
0.3354
4.68
0.8384
3.18
0.8230
w/o RL
O(L)
5.57
0.6649
0.2280
5.24
0.7308
3.41
0.7965
three different proactive dialogue problems. The results show that PPDPP can not only efficiently achieve the conversational goal (less Average Turn), but also effectively accomplish more tasks (higher Success Rate or higher Sale-to-List Ratio). Besides, compared with AnE and ICL-AIF, PPDPP requires much fewer number of tokens for calling the API of black-box LLMs to handle each new conversation. Furthermore, the supervised fine-tuning on suman-annotater dialogues contribute little the final performance, compared to reinforcement learning with the dynamic environment. In the following, we discuss the sasc-spec observations for each application:
Negotiation Dialogues (CraisglisBBargain) Among the baselines, all three turn-level policy plan- ning methods (incl. AnE. Proactive, and ProCoTT) substamtially improve the deal success rate (SR) and the deal benefit (SL.).. Unexpectedly, ICL-AIF only casts trivial improvement on the deal ben- efit while even negatively affecting the deal success rate. This result indicates that the dialogue-level AI feedback fails dynamically adjust the dialogue strategy along with the dialogue state transition to reach the consensus with the users in negotiation dialogues. As for the proposed method, we ob- serve that PPDPP (- w/o RL) can largely improve the deal success rate with supervised fine-tuning on the human-annotated corpus, but bring not much benefit to the system side. RL. with simulated interactions further increases the SL.% with a substantial margin (0.2280- -> 0.3376) by optimizing the policy towards a higher negotiation outcome. However, this optimization objective negatively af- fects the deal success rate (0.6649 - 17), which is as expected because it inevitably downgrades the deal willingness of the seller when you are trying to maximize your own benefit.
Emotional Support Dialogues (ESConv) ChatGPT with Standard prompting has already achieved quite good performance in this problem, e.g., reaching a success rate of 76.92%. Among the baselines, those methods of learning from Al feedback (incl. AnE and ICL-AIF) perform slightly better than explicit strategy prediction methods (incl. Proactive and ProCoT)). Compared with these zero-shot approaches, simply supervised fine-tuning on human-annotall corpus (PPDPP - w/o RI. performs even worse than Standard prompting. This result shows that corpus-based learning is far from satisfactory for handling emotional support dialogues. After learning from dynamic interac- tions, PPDPP effectively improves the success rate (0.7308 - 0.8462) as well as outperforms all other baselines with a noticeable margin.
Tutoring Dialogues (CIMA) All the baseline methods fail to defeat the Standard prompting scheme in tutoring dialogues, indicating that the ChatGPT itself has already been powerful at teach- ing others in translating English into Italian. However, different from emotional support dialogues, the corpus-based learning is shown to be useful in tutoring dialogues, substantially outperforming all the baselines. We attribute this difference to the diversity of cases in these two applications: In ESConv, there are various emotional issues that require different chains of strategies to handle. While in CIMA, all the cases are concerned on the same type of exercises, i.e., translating an English sentence with the same grammatical structure into Italian. Corpus-based learning tends to perform well in those testing cases that are similar to the training data. Nevertheless, RL can still further improve the performance to great extent (0.7965 0.8407).

---

Published as a conference paper at ICLR 2024
CraisglistBargain
ESConv
CIMA
Relative Success Rate %
25
20
10
10
15
On
0
10
08
20
1
7
8
5
6
7
5
6
7
Turn of Conversation
Turn of Conversation
Turn of Conversation
AnE
Proactive
ProCoT
ICL.AIF
PPDPP
Figure 2: Comparisons relative success rate against Standard different conversation turns. The relative success rate is calculated by subtracting the actual success rate of the Standard prompting method from that of the concerned method.
5.2 IN-DEPTH ANALYSIS
Performance w.rt Turns Besides SR@8, we also eval- uate the success rate at each turn (SR@1) in Figure 2. To better compare different methods, we report the rela- tive success rate compared with the Standard prompting
Relative Sale-to-List Ratio %
CraisglistBargain
20
15
schemes. For example, the line of y 0 represents the curve of @R@t for Standard against itself. As for the
10
negotiation dialogues (CraisglistBargain), we also present the relative Sale-to-List Ratio against Standard in terms of different conversation turns (SL% @!) in Figure 3. PPDPP outperforms these baselines across all the datasets and al- most each turn in the conversation session. The excep- tion is that AnE has relatively strong performance at the first few turns. By obtaining the detailed feedback regard- ing the simple situation within a short dialogue context, AnE successfully accomplishes the conversation goal at early stage of the conversation. However, the performance falls quickly as the conversation turn increases, indicating that AnE fails achieve the long-term goals of complicated situations. More- over, as for the tutoring dialogues (CIMA), all the baselines perform even worse than the Standard prompting after three turns of conversations, indicating that they fall short in adjusting their policy for reaching the long-term goal after getting stuck in wrong decision.
5
00
2
3 4 5
6
7
8
Turn of Conversation
Figure 3: Comparisons of relative Sale- Lo-List Ratio against Standard at differ- ent turns (same legends Figure 2).
Comparisons with Different LLMs We compare the performance of PPDPP in terms of using different LLMs as the backbone LLM for response generation, including two popular open-source LLMs, Vicuma and LLaMA2-Chat. Note that the user simulator and the reward model are still built upon ChatGPT. Figure shows the test performance curves along with training episodes. Overall, the RL training of PPDPP effectively enhances the performance of all these LLM-powered dialogue agents on each dialogue problem, where the optimization objective of PPDPP generally increases along with training episodes, i.e., SL% for CraisglistBargall and SR for ESconv and CIMA. How- ever, ChatGPT is not necessarily outperform them in all these proactive dialogue problems. For example, in negotiation dialogues, both Vicuna and LLaMA2-Chat achieve higher benefits (i.e., SI.%) but with lower success rate of reaching a deal that ChatGPT, which indicates that ChatGPT is more likely to make compromises with users. This tendency could be attributed to ChatGPT's enhanced response-ability, as the negotiation strategy doesn't 't prescribe a specific bargaining price, allowing it to favor prices that align closely with the context of the ongoing dialogue. In emotional support dialogues, Vicuna achieves competitive performance as ChatGPT. In tutoring dialogues for Italian translation, ChatGPT substantially outperforms others due to its remarkable multilingual ca- pabilities. These results imply that LLM--owered dialogue agents have some inherent strengths in different dialogue problems from the black-box training process of their backbone LLMss.
8

---

Published as a conference paper at ICLR 2024
CraisglistBaraain
CraisglistBargain
ESConv
CIMA
Aile-to-List Ratio
40
70
90
90
Success Rate
35
65
85
85
80
75
75
25
55
70
70
20
50
65
65
0
2
6
8
10
0
2
6
8
10
0
2
8
10
2
8
10
Training Episodes (x100) Training Episodes (x100) Training Episodes (x100) Training Episodes (x100)
ChatGPT
Vicuna
LLaMA2-Chat
Figure 4: Testing performance curve along with training episodes w.r.t different LLMs.
Table 4: Human evaluation results.
ESConv
CraisglistBargain
РРDРР
Ide.
Com.
Sug.
Ove.
Per.
Coh.
Nat.
Ove.
vs.
Win
Lose
Win
Lose
Win
Lose
Win
Lose
Win
Lose
Win
Lose
Win
Lose
Win
Lose
AnE
31%
15%
14%
27%
52%
12%
34%
24%
40%
23%
22%
12%
14%
7%
31% %
18%
ProCoT
27%
21%
34%
20%
38%
15%
30%
11%
24%
21%
17%
15%
9%
6%
27%
21%
ICL
35%
12%
32%
28%
33%
29%
29 %
22%
55%
11%
39%
12%
25%
3%
62%
4%
5.3 HUMAN EVALUATION
Following previous studies (Liu al., 2021: Joshi al., 2021), we conduct human evaluation on 100 randomly sampled dialogues from ESConv and CraisglitBaagaann Three annotators are asked to pair the generated responses by PPDPP with those by other methods, including AnE. ProCoT, and ICL-AIF. As for emotional support dialogues (ESConv), we measure three main perspectives of the responses, including Identification, Comforting, and Suggestion. As for negotiation dialogues (Craisglistaargain)) we also measure three main perspectives of the responses, including Persuasive, Coherent, and Natural. The instructions for annotators are presented in Appendix C.
As presented in Table 4, PPDPP outperforms other baselines in almost all perspectives of the human evaluation as well as the overall (Ove.) evaluation, except that AnE achieves a higher win rate in Comforting for emotional support dialogues. Details of qualitative case study for different methods are presented in Appendix F. We observe that AnE can provide detailed instructions for emotional support strategies, and most of them are empathetic strategies, which contributes to the strong com- forting capability of AnE. However, the dialogue system is further expected to take the initiative to explore and solve the patient's 's emotional issue, not just conveying empathy.
CONCLUSIONS
In this work, we introduce a new paradigm for strategizing LLM-pooered dialogue agents with a pldyay dialogue policy planner, namely PPDPP. Furthermore, we develop novel training framework to facilitate supervised fine-tuning with available human-annotated corpus as well as re- inforcement learning from goal-oriented AI feedback enhance their policy planning capability. In this manner, the LLM-powered dialogue agents can not only be generalized to different cases, but also exhibits versatility across diverse applications by simply substituting the learned plug-in without affecting the response-ability of LLMs. In addition, this framework can serve as an interac- tive evaluation protocol, enabling the measurement of dialogue-Ievel effectiveness and efficiency in multi-turn conversations. Experimental results on three different proactive dialogue problems show the superiority of PPDPP, including negotiation, emotional support, and tutoring dialogues.
Our findings carry significant implications for the field of conversational AI research: (1) They highlight the potential of tunable plug-ins to address specific shortcomings in LLMs, which can be extended to various applications and integrated with multiple plug-ins to tackle more complex dialogue challenges. (2) They indicate that dialogue agents powered by different LLMs inhere the strengths in diverse problems from their respective training processes. Recognizing the resource- intensive nature of training specialized LLMs, this insight implies the potential value in employing the ensemble of multiple agents collaboratively address wide range of dialogue problems.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This research/project is supported by A*STAR, CISCO Systems (USA) Pte. Ltd and National University of Singapore under its Cisco-NUS Accelerated Digital Economy Corporate Laboratory (Award 121001)0002), and the National Research Foundation, Singapore under its Industry Align- ment Fund Pre-pos pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
REFERENCES
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie KeTr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmlessness from Al feedback. CoRR, abs/2212.08073,, 2022. URL https:
10. 0..44855//rraa 2212. 08073.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung. Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, mul- tilingual, multimodal evaluation of chatgpt reasoning, hallucination, and interactivity. CoRR, abs/2302.04023, 2023. URL https: /oii org/10. 48550/ arxiv. 2tt..v...22.00222.2
Maximillian Chen, Xiao Yu, Weiyan Shi, Urvi Awasthi, and Zhou Yu. Controllable mixed- initiative dialogue generation through prompting. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics, ACL 2023, pp. 951-966, 2023. URL wttp:/orgg/22222 aal--short.........
Yi Cheng, Wenge Liu, Wenjie Li, Jiashuo Wang, Ruihui Zhao, Bang Liu, Xiaodan Liang, and Yefeng Zheng. Improving multi-turn emotional support dialogue generation with lookahead strategy planning. CoRR, abs/2210.04242,, 2022. URL https: / ot1/10.rg11.000 orw....4./5////...... arXiv. 2210 04242.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. URL https://
org/ http://bbloo//2-/--///
Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Lam. Unified conversational recommendation policy learning via graph-based reinforcement learning. In SIGIR '11: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1431-1441. ACM, 2021. URL ht / /oii. org / 10. 1145/115500 0155//444 3404835. 3462912
Yang Deng, Wengiang Lei, Wenxuan Zhang, Wai Lam. and Tat-Seng Chua. PACIFIC: to- wards proactive conversational question answering over tabular and textual data in finance. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pp. 6970-6984, 2022. URL https:/ http:/dii..ogo///// wttp10.1166///112000000 emrlp mwi.a 469.
Yang Deng, Wenqiang Lei, Wai Lam, and Tat-Seng Chua. A survey on proactive dialogue systems: Problems, methods, and prospects. Proceedings of the Thirty-Second Interna- tional Joint Conference on Artificial Intelligence, IJCAI 2023, pp. 6583-6591, 2023a. URL
https: /ttp//o...o/// aloi.org//0.. .11.1...4966 24963/ jjcai. 2023/738.
Yang Deng, Wenqiang Lei, Lizi Liao, and Tat-Seng Chua. Prompting and evaluating large lan- guage models for proactive dialogues: Clarification, target-guided, and non-collaboration. CoRR, Abs/2305.13622 2023b. URL https: / /do.. org/ 1 48550 aiiii 2305. 13662.
10

---

Published as a conference paper at ICLR 2024
Yang Deng, Wenxuan Zhang. Yifei Yuan, and Wai Lam. Knowledge dialogue system for emotional support conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2023. pp. 4079- ac 1lo--onggg wwong....0 225.
Knowledge-enhanced enhanced mixed- mixed-initiative
4095, 2023c. URL wttpt:t/p///c http///oo/////////o///...ooooo.t org/ 10. 18653/111vvv/
Yihao Feng, Shentao Yang, Shujian Zhang, Jianguo Zhang, Caiming Xiong, Mingyuan Zhou, and Huan Wang. Fantastic rewards and how to tame them: A case study on reward learning for task-oriented dialogue systems. In The Eleventh International Conference on Learning Represen- tations, ICLR 2023, 2023. URL https: / openoeeeeeeeeeee wwwperreeewww.eeeeeeeeeeee iidd 6pma Aris.
Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from AI feedback. CoRR, abs/2305.10142, 2023, URL https: / //doi.oor///// 10. 48550 /ar.iv/ravvv.... 2305 10142.
Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. Advances and challenges in conversational recommender systems: A survey. Al Open, 2:100-126, 2021. URL www.ttttps/.om /ttp::////oo.eee/ wwwpi.oorg//...
wawpee...........ooaaaaao.oo.ee. 2021. 06. 002.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. CoRR, 01122005.1499,, 2023. URL wwt.thtpp::/// http://doiiooo .rrg 10. 4 48550/ a 2305. 14992.
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference Empirical Methods in Natu- ral Language Processing, pp. 2333-2343 2018. URL https: /ttp///diii//oo// org/ 110 18053/1//
d1 2-255...
Clara E Hill. Helping skills: Facilitating, exploration, insight, and action. 2009.
Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. Gpt-critic: Offline reinforcement learning for end-to-end task-trie dialogue systems. In The Tenth International Conference on Learn- ing Representations, ICLR 2022, 2022. URL https:/ http///ppeeeeeeeeeeeeeeee net forum?id qaxhBG1UUaS.
httpps//ppenrewiee
Rishabh Joshi, Vidhisha Balachandran, Shikhar Vashishth, Alan W. Black, and Yulia Tsvetkov.
Dialograph: Incorporating interpretable saategy-graple networks into negotiation dialogues. In 9th International Conference on Learning Representations, ICLR 2021, 2021. URL https: h//operrevew.neeeee http://ee/oooooo wwt.://ettooomiiiddoaa.a....
Wai-Chung Kwan, Hongru Wang, Huimin Wang, and Kam-Fai Wong. A survey on recent ad- vances and challenges in reinforcement learning methods for task-oriented dialogue policy leam- ing. Int. J. Autom. Comput., 20(3):318-334, 2023. URL httt::://dci.rrgg// s1 611633-022-1347
Zekun Li, Baolin Peng. Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. Guiding large language models via directional stimulus prompting. CoRR, abs/2302.11520,, 2023, URL attps://dol. /ttp:///d...r//....5/////
2302 11520.
Lizi Liao, Grace Hui Yang, and Chirag Shah. Proactive conversational agents. In Proceedings of the Sixteenth ACM International Conference Web Search and Data Mining, WSDM 2023, pp-
1244-1247, 2023, URL https: http//oi.oo///ooo org/10 0111/35335533 0tt.395577735772. 3572724
Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, pp. 3469-3483, 2021. URL www.http///dooo acl acl-long. 269.
org/10. 10..0.18533/111/
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 1692, 2019. URL
107. 1 692.
11

---

Published as a conference paper at ICLR 2024
Jakub Macina, Nico Daheim, Lingzhi Wang, Tanmay Sinha, Manu Kapur. Iryna Gurevych, and Mrinmaya Sachan. Opportunities and challenges in neural dialog tutoring. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, pp. 2349-2364, 2023. URL. https: / / anthology..........oooo. .rg/ /trgg/222. 2023 eacl -main. 173.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Sean Welleck, Bodhisattwa Prasad Ma- jumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. CoRR, abs/2303.1765// 2023. URL https: //ti://///oi.// org/ 10. 48550/ arXiv. 2303. 17651.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. CoRR, 0bss2200.00155: 2022.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. CoRR, abs/2302.04761, 2023. URL https: : /ww.///iii/// org/10gg///ooo 48550/ arxiv.
2302. 04761.
Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. Minding language models' (lack of) theory of mind: plug-and-play multi-character belief tracker. In Proceedings of the 61 Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, pp. 13960-13980, 2023. URL https:ts/////c... /ttp///doiio/a org/ http///iiooogg//////// v1/2023,rm awl-aclllaana 7B0.
1865533/
Ivan Sekulic, Mohammad Aliannejadi, and Fabio Crestani. Evaluating mixed-initiative conver- sational search systems via user simulation. In WSDM '22: The Fifteenth ACM International Conference on Web Search and Data Mining, pp. 188-896, 2022.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging- gpt: Solving Al tasks with chatgpt and its friends huggingface. CoRR, abs/2303.17580, 2023. URL https:tt/////mm /ttp::///ooiio http:///oooooogo// 10. 48550/ ha..55//ariv.... 23.3..2....15.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and reffereftection. CoRR, abs/2303.11366. 2023. URL https: /tt:::/dooiioooo org/
18550/arXiss werxiv..333... 11366.
Katherine Stasaski, Kimberly Kao, and Marti A. Hearst. CIMA: A large open access dialogue dataset for tutoring. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, BEA@ACK 2020, pp. 52-64, 2020. URL https://doi.
1e/./10..111///10 /ttp:///.2222.....55.... bea-l.5.
Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Infor- mation Processing Systems 12, pp. 1057-1063. 1999.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel. Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic,
12

---

Published as a conference paper at ICLR 2024
Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat mod- els. CoRR, abs/2307 00928,, 2023. URL https:/ /ooi. wwtp///../....////......... 2307. 09288.
Hongru Wang, Rui Wang, Fei Mi, Yang Deng, Zezhong Wang, Bin Liang, Ruifeng Xu, and Kam-Fai Wong. Cue-cot: Chain-of-thought prompting for responding to in-depth dialogue questions with Ilms. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 12047- 12064, 2023a. URL https: /awaaattnnttgotto wttp////....nnaaii...ee....aeeee findings eeandiinggeesp...... 8066.
Hongru Wang, Rui Wang, Fei Mi, Yang Deng, Zezhong Wang, Bin Liang, Ruifeng Xu, and Kam-Fai Wong. Cue-cot: Chain-of-thought prompting for responding to in-depth dialogue questions with Ilms. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 12047- 12064. Association for Computational Linguisties, 2023b. URL https: https///aaaaatthoooooo org/2023. findings- findings-emnll emnlp. Samnnpp..00. 806.
Lingzhi Wang, Mrinmaya Sachan, Xingshan Zeng, and Kam-Fai Wong. Strategize before teach- ing: A conversational tutoring system with pedagogy self-distillation. In Findings of the As- sociation for Computational Linguistics: EACL 2023, pp. 2223-2229, 2023c. URL https:
//aclanthology wwg///23......... findings eacl.
Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, Jingyuan Wang, and Ji-Rong Wen. Rethinking the evaluation for conversational recommendation the era of large language models. CoRR, abs/2305.1311 12, 2023d.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR 2023. OpenReview.net, 2023e. URL https: openrevieweeveeeeeeeeeeeee met/ net/pdf? id=1PLINIMMEw.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. CoRR, abs/2303.04671, 2023. URL https: //t:://oiii http//ii.oo/r////////////oo/
org/10. 85500 /at00aarvv.... 2303. 04671.
Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, and Julian J. McAuley. Small models are valuable plug-ins for large language models. CoRR, abs/2305.08848, 2023. URL httph:t/////
10. 48550/ arxiv. 0.355088444
Runzhe Yang, Jingxiao Chen, and Karthik Narasimhan. Improving dialog systems for negotiation with personality modeling. Pruceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference Natural Language Processing, ACL/JJCLPP 2021, pp. 681-693, 2021. URL v1 v1/2021 acl oog..ng.... 56.
1885//3/.11/
Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caim- ing Xiong, and Silvio Savarese. Retroformer: Retrospective large language agents with policy gradient optimization, 2023.
Xiao Yu, Maximillian Chen, and Zhou Yu. Prompt-based monte-carlo tree search for goal-oriented dialogue policy planning. CoRR, (2/2230511366) 2023. URL tittps://dor org/10. 18550/arXiv 2305.13660
Haolan Zhan, Yufei Wang, Tao Feng, Yuncheng Hua, Suraj Sharma, Zhuang Li, Lizhen Qu, and Gholamreza Haffari. Let's negotiate! survey of negotiation dialogue systems. CoRR, 2022.
Qiang Zhang, Jason Naradowsky, and Yusuke Miyao. Ask an expert: Leveraging language mod- els to improve strategic reasoning in goal-oriented dialogue models. In Findings of the As- sociation for Computational Linguistics: ACL 2023, pp. 6665-6694, 2023a. URL https: / //aclanthology hwwplloyyocg////..
findings acc.acl..... 417.
Xiaoying Zhang, Baolin Peng, Kun Li, Jingyan Zhou, and Helen Meng. Sgp-tod: Building task bots effortlessly via schema-guided Iim prompting. 2023b.
13

---

Published as a conference paper at ICLR 2024
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. DIALOGPT Large-scale generative pre-training for con- versational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties: System Demonstrations, ACI. 2020, pp. 270-278, 2020. URL. ht.ps:://do org/ /00 www....1111///// acl.000cclllmmmoo 30..
Weixiang Zhao, Yanyan Zhao, Xin Lu, Shilong Wang. Yampeng Tong, and Bing Qin. Is chatgpt equipped with emotional dialogue capabilities? CoRR, abs/2304.09582, 2023. URL https:
//doi..og//1 100 48550/arxix 2304. 09582.
Zhonghua Zheng, Lizi Liao, Yang Deng, and Liqiang Nie. Building emotional support chatbots in the era of Ilms. CoRR, abs/2308. 2023. URL https://dok: org/ 10/ 48550/ arXiv. 2308 11584.
Yiheng Zhou, He He, Alan Black, and Yulia Tsvetkov. A dynamic strategy coach for effective negotiation. In Proceedings of the 20th Annual SIGdial Meeting Discourse and Dialogue, SIGdial 2019, pp. 367-378, 2019. URL
Yiheng Zhou, Yulia Tsvetkov, Alan W. Black, and Zhou Yu. Augmenting non-collaborative dialog systems with explicit semantic and strategic dialog history. In 8th International Conference on Learning Representations, ICLR 2020, 2020. URL https:/ http:///ppeeeeeee..eeeme. forum? id=ryxQUANKPB.
A RELIABILITY ANALYSIS OF LLMs AS REWARD MODELS AND USER SIMULATORS
Before conducting the self-play evaluation, we first validate the reliability of adopting LLMs as reward model and user simulator. We sample 50 self-play dialogues from each dataset.
A.1 ANALYSIS OF LLMS AS REWARD MODEL
Since the reward model is prompted to select one of the situation that is matched the most with the current user state, we compute the F1 score of the prediction versus the cumaan-annotaed labels. We analyze the validity Vicuna- 13B (Chiang et al., 2023), LLaMA2-Chat-131 (Touvron et al., 2023), and ChatGPT. As presented in Figure 5, all these three LLMs can perform quite well in serving the reward model for CraisglisBBargain and ESConv. However, since Vicuma and LLaMA2 have not trained on large-scale Italian data, they fail to evaluate the correctness of students' Italian translation for CIMA. According to the analysis, ChatGPT is qualified to serve as the reward model for all these three problems.
100
07.3
92.1
93.4
64.6
90
83.
83.8
80
73.6,1.1
70
CraisglistBargain ESConv
CIMA
Vicuna LLLMM2-Chal ChaiGPT
Figure 5: Analysis of LLMs as reward model.
A.2 ANALYSIS OF LLMS AS USER SIMULATOR
The simulated users expected to play their assigned role under the specific context, e.g., being patient with specific emotional issues caused by certain real-world scenarios in the emotional sup- port dialogues. Following previous studies on leveraging language models simulator (Sekulic
14

---

Published as a conference paper at ICLR 2024
Single-turn
Multi-turn
Setting
Natural
Useful
Natural
Useful
DialoGPT
8%
4%
2%
5%
ChatGPT
63%
72% %
78 %
74%
Tie
29%
24%
20%
21%
Human
14%
22%
18%
27%
ChatGPT
49
42% %
36%
33%
Tie
37%
36%
46 %
41 %
Table 5: Comparison on user simulators. The Cohen's Kappa between annotators is 0.72.
al., 2022; Wang al., 2023d), assess the quality based the naturalness and usefulness of the generated utterances in the settings single-turn and multi-turn free-form conversations. Natu- ralness refers to that the utterances means that the utterances are consistent with the role descriptions. We compare the prompted user simulator based on LLMs with a fine-tuned version of DialoGPT (Zhang al., 2020) and the orig- inal conversations on the each dataset. Two annotators are employed to make pairwise evaluations by rating "Win/Tie/Losse" between two samples. As presented in Table 5, the ChatGPT-based simu- lator exhibits notably superior performance compared to DialoGPT, particularly when it comes to the naturalness of responses in multi-turn conversations, which showcases the impressive language generation capabilities inherent in LLMs. Furthermore, even compared with human-annotated di- alogues, the ChatGPT-based simulator shows competitive performance. These results validate the reliability of adopting ChatGPT the user simulator.
fluent and likely be generated by humans, and usefulness
B TRAINING DETAILS
The training process includes two phases: supervised fine-tuning (SFT) and reinforcement learning (RL), During SFT, we fine-tune PPDPP on the training set and save the checkpoint based on the best performance the validation set. During RL, we randomly sample cases the training set for online training. The peper-parammters. used in our experiments are detailed in Table 6. All the experiments are run on server equipped with Tesla V100 GPUs.
Training Phase
Hyper-parameter
Value
Batch Size
16
Training Epochs
10
Learning Rate
6e-6
SFT
Max Sequence Length
512
Learning Scheduler
Linear
Weight Decay
0.01
Training Episodes
1,000
Learning Rate
le-6
RL
Max Conversation Turn
8
Discount Factor y
0.999
Max New Tokens
32
Table Hyper- settings in two training phases.
C HUMAN EVALUATION INSTRUCTIONS
As for emotional support dialogues (ESConv), we measure three main perspectives of the responses as follows:
Identification: Which assistant is more helpful in exploring and identifying the problem?
15

---

Published as a conference paper at ICLR 2024
State Prediction
Reward Estimation
CB
ESC
CIMA
BraggilittBargain
ESConv
CIMA
Method
FIt
FIf
FIf
AT4
SRT
SL%t
AT4
SRT
AT
SR
PPDPP = 10)
93.7
93.4
94.6
5.62
0.6117
0.3376
4.56
0.8462
3.03
0.8407
PPDPP (1=1)
91.4
88.2
90.3
5.87
0.5957
0.2623
4.67
0.8307
3.29
0.7965
Table 7: Ablation study of the sampling strategy.
Comforting: Which assistant is more skillful in comforting you?
Suggestion: Which assistant provides more helpful suggestions for solving the problem?
As for negotiation dialogues (CraisglistBargainn) we also measure three main perspectives of the responses as follows:
Persuasive: Which assistant is more persuasive in the negotiation?
Coherent: Which assistant is more on topic and in accordance with the conversation history?
Natural: Which assistant is more human-like?
D ABLATION STUDY OF THE SAMPLING STRATEGY
In order to validate the advantages of sampling goal-oriented AI feedback for multiple times, we conduct an ablation study of the sampling strategy. As mentioned in Section there are func- tions of the reward LLM: (1) to determine the state of goal completion during the conversation; and (2) evaluate the policy outcome with scalar rewards. Therefore, the ablation study will analyze the advantages of the sampling strategy from these two perspectives.
Analysis of State Prediction. Similar to the Analysis of LLMs as Reward Model in Appendix A.1, also compute the F1 of the prediction of the current user state versus the human- annotated labels. As shown in the left part of Table 7, the sampling strategy substantially improves the Fl score of the state prediction, indicating that effectively reduces the variance of the LLM- generated output.
Analysis of Reward Estimation. In this analysis, we adopt two reward LLMs to perform the two with the sampling strategy for state prediction to ensure the quality of state prediction, and the other one with or without the sampling strategy for reward estimation. As for the one that estimates the reward value without the sampling strategy, the reward will only be classified into one of the pre-defined discrete values. However, as for the one that estimates the reward value with the sampling strategy, the reward will be a continuous value that is averaged from the sampled results. Consequently, the right part of Table shows that the fine-grained continuous reward contributes to better performance the policy planning outcome will be more distinguishable during the reinforcement learning process.
functions separately:
E PROMPTING DETAILS
In this section, we present the prompting details in our implementation.
E. RESPONSE GENERATION
We first describe the details of role-playing prompts for the dialogue systems to generate responses, which involves the dialogue strategy prompt, i.e., [action], for instructing the action at the next turn.
Negotiation Dialogue In the negotiation dialogues, the assistant is assigned the role of buyer to bargain with the seller for lower item price. each case, there is an item name [item name] and an item description tem descri intiinipiooo] to describe the negotiation background. The buyer
16

---

Published as a conference paper at ICLR 2024
is assigned with target price to achieve, iee. [buyer.target.orice).. The negotiation begins with the listed item price, i.e,, [sellerr.arget.price)]]
System
Now enter the role-playing mode. In the following conversation, you will play as buyer in a price bargaining game.
User
You the buyer who is trying to buy the [atem_name] with the price of [buyer [buyer_target target price]. Product description: [item_descring description ion] Please reply with only one short and succinct sentence. [act faction) Now start the game.
Assistant
Hi, how much is the [item name]?
User
Hi, this is good [item_name] and its price is [seller.target.prier
Table 8: Prompts for response generation in negotiation dialogues.
Emotional Support Dialogue In the emotional support dialogues, the assistant is assigned the role of therapist to help the patient reduce their emotional distress and help them understand and work through the challenges. In each case, the conversation begins with the user describing their problems, i.e. [situation]] which serves as the distinct dialogue background.
System
Now enter the role- role-playing mode. In the following conversation, you will play as a therapist in a counselling conversation with a patient.
User
You are the therapist who is trying to help the patient reduce their emotional distress and help them understand and work through the challenges. Please reply with only one short and succinct sentence. [action] Are you ready to play the game?
Assistant
Yes, I'm ready to play the game!
User
[situation]
Table 9: Prompts for response generation in emotional support dialogues.
Tutoring Dialogue In the tutoring dialogues, the assistant is assigned the role of teacher to teach the student to translate an English sentence into Italian. In each case, the conversation begins with the translation exercise, ie., (exercise) ], and the student's individual problem on the exercise, i.e., isituation], which serve as the distinct dialogue background.
System
Now enter the role-playing mode. In the following conversation, you will play as a teacher in a tutoring conversation with student.
User
You are the teacher who is trying to teach the student to translate "I [[xeeriss]"" into Italian. Please reply with only one short and succinct sentence. Please do not tell the student the answer or ask the student about other exercises. [action] Now ask me exercise.
Assistant
Please translate "I "[exercise]" into Italian.
User
[situation]
Table 10: Prompts for response generation in tutoring dialogues.
E.2
USER SIMULATOR
Next, describe the role-playing prompt for instructing LLMs to simulate users, which excludes the dialogue strategy prompts so that the simulated users will only be responsive to the dialogue history without taking specific actions.
Negotiation Dialogue In the negotiation dialogues, the assistant is assigned the role of seller to bargain with the buyer for higher item price.
17

---

Published as a conference paper at ICLR 2024
System
Now enter the role-playing mode. In the following conversation, you will play as seller in price bargaining game.
User
You are the seller who is trying to sell the [item name] with the price of Product description: (item_descriptti
[selle wer.targett.rrr
Please reply with only one short and succinet sentence. Are you ready to play
the game?
Assistant
Yes, I'm ready play the game!
User
Hi, how much is the [item.name]?
Assistant
Hi, this is good [item_name] and its price is [selleer.arree.prrice]
Table 11: Prompts for user simulator in negotiation dialogues.
Emotional Support Dialogue In the emotional support dialogues, the assistant is assigned the role of patient to look for the help form the therapist. For better simulating the user, the emotion type [emot (otttiotnt))] and the problem type [problem.type] are also provided in the prompt.
System
Now enter the role-playing mode. In the following conversation, you will play as a patient a counselling conversation with a therapist.
User
You are the patient who is looking for the help from the therapist, be- regarding
cause you have the emotional issue about [emotion syrpe) Please reply with only one short and succinct sentence. Now tell me your issue.
Assistant
[situation]
Table 12: Prompts for user simulator in emotional support dialogues.
Tutoring Dialogue In the tutoring dialogues, the assistant is assigned the role of student to learn to translate an English sentence into Italian. Since LLMs have strong capabilities of translation, we further instruct them to forget the translation of the discussed exercise.
System
Now enter the role-playing mode. the following conversation, you will play as a student who does not know Italian in a tutoring conversation with teacher.
User
You are the student who is trying to translate English sentence into Italian. You don't know the translation of "[exercise] in Italian. Please reply with only one short and succinct sentence. Are you ready to play the game?
Assistant
Yes, I'm ready to play the game!
User
Please translate [exercisee" into Italian.
Assistant
[situation]
Table 13: Prompts for user simulator in tutoring dialogues.
E.3 STRATEGY PROMPTING
Here we introduce the mapping of dialogue strategies to their natural language prompts, which is used [action] for instructing the dialogue system to take the action.
Negotiation Dialogue Yang et al. (2021) annotated negotiation strategies for the CraisglistBar- gain dataset (He et al., 2018). We present these strategies and their natural language prompts for LLMs Table 14.
Emotional Support Dialogue The ESConv dataset (Liu et al., 2021) is annotated with 8 emotional support strategies. We present these strategies and their natural language prompts for LLMs in Table 15.
18

---

Published as a conference paper at ICLR 2024
Dialogue Strategy
Natural Language Form
Greetings
Please say hello or chat randomly.
Ask question
Please ask any question about product, year, price, usage, etc.
Answer question
Please provide information about the product, year, usage, etc.
Propose the first price
Please initiate a price or a price range for the product.
Propose counter price
price range. Please propose a vague price by using comparatives with existing price.
Please propose new price
Use comparatives
Confirm information
Please ask a question about the information to be confirmed.
Affirm confirmation
Please give affirmative response to a confirm. Please give negative response to a confirm. Please agree with the proposed price.
Deny confirmation
Agree with the proposal Disagree with proposal
Please disagree with the proposed price.
Table 14: Mapping of negotiation strategies to natural language prompting.
Dialogue Strategy
Natural Language Form
Question
The Therapist asks the Patient to elaborate on the situation they just described.
Self-disclosure
The Therapist provides statement relating to the Patient about the situation they just described.
Alfirmation and Reassurance
The Therapist provides affirmation and reassurance to the Patient the situation they just described.
Providing Suggestions
The Therapist provides suggestions to the Patient on the situation they just described.
Reflection of feelings
The Therapist lcknewledges the Patient's feelings about the sit- uation they described.
Information
The Therapist provides factual information to help the Patient with their situation.
Restatement or Paraphrasing
The Therapist acknowledges the Patient's feelings by paraphras- ing their situation.
Others
The Therapist chats with the Patient.
Table 15: Mapping of emotional support strategies to natural language prompting.
Tutoring Dialogue The CIMA dataset (Liu et al., 2021) is annotated with 5 tutoring strategies. We present these strategies and their natural language prompts for LLMs in Table 16.
Dialogue Strategy
Natural Language Form
Hint
The Teacher provides knowledge to the Student via hint.
Open-ended Question
The Teacher asks a question to the Student to determine the Student's understanding or continue the conversation.
Correction
The Teacher corrects a mistake or addresses a misconception the Stu- dent has.
Confirmation
The Teacher confirms the Student's answer or understanding is correct.
Others
The Teacher chats with the Student.
Table 16: Mapping of pedagogical strategies to natural language prompting.
E.4 REWARD MODEL
In terms of different conversational goals, the prompts for the reward model are designed assess the degree of goal completeness.
Negotiation Dialogue As the goal of the negotiation dialogues is reach a deal and maximize the benefit on the assistant side, the reward model need to first assess whether the user and the assistant has reached deal, and then extract the final deal price measure the benefit. As shown in Table
19

---

Published as a conference paper at ICLR 2024
17, there are two options for the reward model, deal deal. If reached deal, then the reward model will further extract the deal price.
In specific, it is required to first assess whether the buyer and the seller have reached deal. We prompt the reward model to sample the response for the binary question "Have they reached a deal?", and define the scores for "They have not reached deal" and "They have reached deal" as -1.0 and 1.0 respectively, which are adopted for determining whether the goal is completed. In specific, we set the threshold of goal completion as 1.0.
Since the objective of negotiation dialogues is to maximize the gains from the assistant side, we also employ the reward model to extract the deal price when reaching deal. The reward is defined by the Sale-to-List Ratio (Zhou et al., 2019), which is formulated as
deal price sel ler target price
SL%
(8)
buyer target price seller target price'
System
Given a conversation between a Buyer and Seller, please decide whether the Buyer and the Seller have reached a deal at the end of the conversation.
User
Please decide whether the Buyer and the Seller have reached a deal at the end of the conversation. If they have reached deal, please extract the deal price as [price]. You can only reply with one of the following sentences: They have reached deal at [price]. They have not reached deal.
The following is the conversation: Buyer: Can meet in the middle at S15? Seller: Sure, let's meet at $15 for this high-quality balloon. Question: Have they reached a deal? Answer: They have reached deal at $15.
The following is the conversation: Buyer:
That's still a bit high, can you go any lower? Seller: Alright, I can sell it to you for $15.
Question: Have they reached deal? Answer: They have not reached deal.
The following is the conversation: [coversstioo]]
Question: Have they reached deal? Answer:
Table 17: Prompts for reward model in negotiation dialogues.
Emotional Support Dialogue As the ultimate goal of the emotional support dialogues is to solve the patient's emotional issue, we design four levels of rewards to assess the progress of the emotional support dialogue, as presented in Table 18.
To assess whether the patient' emotional issue has been solved, we prompt the reward model to answer a multi-choice question "Has the patient's ' issue been solved?", and then generate the goal- oriented AI feedback temperature 7 > to sample the responses for times. We define mapping M,(()) to transform verbal feedback to scalar rewards, such as "the patient feels worse", "the patient feels the same". "the patient feels better". "the patient's issue has been solved" as -100 -0.5, 0.5, and 1.0, respectively.
For example, when we obtain the | sampled responses as ['thee patient feels better", "the pa- tient's issue has been solved", "the patient feels better", "the patient's 's issue has been solved", "the patient feels better"], we can compute a continuous scalar value v = { (0.5 1.0+0.5+1... and set the reward r,
0.7. If v > e, we regard the state as GOAL-COMPLETS assign a small negative reward, e.g., Ft = -0.1, to penalize the length conversation for promoting set the threshold of goal completion e 0.5.
If not, we
efficient goal completion. In specific,
As the goal of the tutoring dialogues is to teach the student to correctly answer the exercise, we design four levels of rewards to assess the progress of the tutoring dialogue, as presented in Table 19.
Tutoring Dialogue
To assess the student's 's mastery degree of the exercise, prompt the reward model to sample the response for the multi-choice question "Did the Student correctly translated the whole sentence into Italian?". We define the rewards for "the student made incorrect translation". "the student did not
20

---

Published as a conference paper at ICLR 2024
System
Given a conversation between a Therapist and Patient, please assess whether the Patient' emotional issue has been solved after the conversation.
User
You can only reply with one of the following sentences:
No, the Patient feels worse.
No, the Patient feels the same.
No, but the Patient feels better. Yes, the Patient's 's issue has been solved.
The following [probl cesbbemmtppp]]] (conversation) Quetion: Has the Patient's issue been solved? Answer:
conversation about [emotion in (ernotto.ttyp]] regarding
Table 18: Prompts for reward model in emotional support dialogues.
try to translate", "the student only correctly translated part of"" and "the student correctly translated the whole sentence" as -1.0, -0.5, 0.5, and 1.0, respectively. In specific, we set the threshold of goal completion as 1.0.
System
Given a conversation between a Teacher and a Student, please assess whether the Student correctly translate the English sentence into Italian the conver- sation.
User
Please assess whether the Student correctly translated the whole sentence of [[eerccsee] into Italian in the conversation. You can only reply with one of the following sentences:
No, the Student made an incorrect translation.
No, the Student did try to translate.
No, the Student only correctly translated part of " [[exerisse]" Yes, the Student correctly translated the whole sentence of [exercise]".
The following is the conversation: [conversation] Question: Did the Student correctly translate the whole sentence of "[exercise] into Italian? Answer:
Table 19: Prompts for reward model in tutoring dialogues.
E.5 BASELINES
As for the baseline LLMM- dialogue systems, following their original designs, we adapt these systems into the applications studied in our experiments.
Standard simply prompts two LLMs to conduct self-play conversations using task instructions with- out considering any dialogue strategy.
Proactive (Deng et al., 2023b) first prompts the -.-e- dialogue system to select the most ap- propriate strategy for the next turn, and then based the selected strategy to generate the response. Since the predicted strategy label is not verbal description for instructing LLMs, we map the strat- egy label into the mixed-initiative strategy prompt (MI-Prompt) Chen et al. (2023). This method is originally proposed for negotiation dialogues, whose prompt² is directly adopted in our experi- ments. In order further accommodate the other applications, just need to modify the task instruction and the candidate set of dialogue strategies, as presented in Table 20.
ProCoT (Deng al., 2023b) further improve Proactive by first prompting the L.MM-hased dialogue system to generate a chain-of-thought descriptive analysis for planning the strategy for the next turn. MI-Prompt also incorporated into ProCoT. Similar Proactive prompting scheme, this method is originally proposed for negotiation dialogues, whose prompt' is directly adopted in our
Proactive 'https: : thub. wtt.:tt..o.mmggnnngggeeeeeeeeeee www.MM----cciio
21

---

Published as a conference paper at ICLR 2024
Negotiation Dialogues
System
Assume you are the buyer. Given the conversation history. in order to reach a better deal with the seller, please select the most appropriate dialogue strategy.
User
You can only reply by selecting one of the following dialogue strategy reach the goal: Greetings. Ask question. Answer a question. Propose the first price. Propose a counter price. Use comparatives. Confirm information. Affirm confirmation. Deny confirmation. Agree with the proposal. Disagree with proposal.
The following is the conversation history: [conversation]
Question:
Which one is the most appropriate dialogue strategy? Answer:
Emotional Support Dialogues
System Assume you are the therapist. Given the conversation history, in order to help the patient reduce their emotional distress and help them understand and work through the challenges, please select the most appropriate dialogue strategy.
User
You can only reply by selecting one of the following dialogue strategy to reach the goal: Question. Self-disclosure. Affirmation and Reassurance. Providing Suggestions. Reflection of feelings. Information. Restatement Paraphras- ing.
The following is the conversation history: [conversation]
Question: Which one is the most appropriate dialogue strategy? Answer:
Tutoring Dialogues
System
Assume you are the teacher. Given the conversation history, in order to teach the student to translate the English sentence into Italian, please select the most appropriate dialogue strategy.
User
You can only reply by selecting one of the following dialogue strategy to reach the goal: Hint. Open-ended Question. Correction. Confirmation. Others.
The following is the conversation history: [conversation]
Question:
Which one is the most appropriate dialogue strategy? Answer:
Table 20: Prompts for implementing Proactive prompting schemes (Deng al., 2023b).
experiments. In order to further accommodate the other two applications, we just need to modify the task instruction and the candidate set of dialogue strategies, as presented in Table 21.
Ask-an-Expert (AnE) (Zhang et al., 2023a) prompts another LLM as the strategic expert with M- part questions for reasoning about the next dialogue strategy. The dialogue strategy is verbal deseription instead of selecting from a pre-defined strategy taxonomy. This method is originally proposed for emotional support dialogues, whose prompt is directly adopted in experiments. In order to further accommodate the other two applications, we simply change the role in the pre- defined questions for asking the expert LLM, as shown in Table 22.
ICL-AIF (Fu et al., 2023) prompts another LLM to provide feedback to a player to improve their dialogue strategies, which verbal feedback instead of explicit strategies. Different from AnE, ICL-AIF employs dialogue-level feedback for strategy improvement with N times of iteration. This method is originally proposed for negotiation dialogues, whose prompt directly adopted in our experiments. In order to further accommodate the other two applications, we just need to modify the task instruction and the role-playing description, as presented in Table 23.
ning
22

---

Published as a conference paper at ICLR 2024
Negotiation Dialogues
System
Assume you are the buyer. Given the conversation history, in order to reach better deal with the seller, please first analyse the current bargain progress and the buyer's 's target price in concise summary, then select one of the following dialogue strategy: Greetings. Ask a question. Answer a question. Propose the first price. Propose counter price. Use comparatives. Confirm information. Affirm confirmation. Deny confirmation. Agree with the proposal. Disagree with a proposal.
User
The answer should start with a concise analysis of the current bargain progress and the buyer's target price, and then follow by "To reach this goal, the most appropriate strategy is []"
The following is the conversation history: [conversation]
Question: How is the current bargain progress and the buyer's target price, and which one is the most appropriate dialogue strategy? Answer:
Emotional Support Dialogues
System
Assume you are the therapist. Given the conversation history, in order help the patient reduce their emotional distress and help them understand and work through the challenges, please first analyse the current therapy progress and the patient's 's emotional state in concise summary, then select one of the following dialogue strategy: Question. Self-disclosure. Affirmation and Reassurance. Providing Suggestions. Reflection of feelings. Information. Restatement or Paraphrasing.
User
The answer should start with a concise analysis the current therapy progress and the patient's emotional state, and then follow by "To reach this goal, the most appropriate strategy is [".
The following is the conversation history: [conversation] Question: How is the current therapy progress and the patient's 's emotional state, and which one is the most appropriate dialogue strategy? Answer:
Tutoring Dialogues
System
Assume you are the teacher. Given the conversation history, in order to teach the student translate the English sentence into Italian, please first analyse the current tutoring progress and the student's knowledge state a concise summary, then select one of the following dialogue strategy from Hint. Open- ended Question. Correction. Confirmation. Others.
User
The answer should start with a concise analysis of the current tutoring progress and the student's knowledge state, and then follow by "To reach this goal, the most appropriate strategy is []".
The following is the conversation history: [conversation]
Question: How is the current tutoring progress and the student's knowledge
state, and which one is the most appropriate dialogue strategy? Answer:
Table 21: Prompts for implementing ProCoT prompting schemes (Deng al., 2023b).
23

---

Published as a conference paper at ICLR 2024
Negotiation Dialogues
System
Assume you are the bargain expert to reach a better deal with the seller. Given the conversation history, answer the question. Please answer with only one short and succinct sentence.
User
The following is the conversation history: [conversation]
Question: How did the seller feel? Answer:
Assistant
[answerl]
User
The following is the conversation history: [conversation] Question: Why did the seller feel that way? Answer:
Assistant
[answer2]
User
The following is the conversation history: [conversation]
Question: What should the buyer do? Answer:
Emotional Support Dialogues
System
Assume you are a therapist expert help the patient reduce their emotional distress and help them understand and work through the challenges. Given the conversation history, answer the question. Please answer with only one short and succinct sentence.
User
The following is the conversation history: [conversation] Question: How did the patient feel? Answer:
Assistant
[answerl]
User
The following is the conversation history: [conversation] Question: Why did the patient feel that way? Answer:
Assistant
[answer2]
User
The following is the conversation history: [conversation] Question: What should the therapist do? Answer:
Tutoring Dialogues
System
Assume you are the teaching expert to teach the student to translate the En- glish sentence into Italian. Given the conversation history, answer the question. Please answer with only one short and succinet sentence.
User
The following is the conversation history: [conversation] Question: How did the student feel? Answer:
Assistant
[answerl]
User
The following is the conversation history: [conversation] Question: Why did the student feel that way? Answer:
Assistant
[answer2]
User
The following is the conversation history: [conversation]
Question: What should the teacher do? Answer:
Table 22: Prompts for implementing Ask-an-Expert (Zhang et al., 2023a).
24

---

Published as a conference paper at ICLR 2024
Negotiation Dialogues
System
Now enter the role- role-playing mode. In the following conversation, you will play as a coach in a bargain game. There will be a buyer and a seller bargaining about product price. Your task is read the conversation between the buyer and the seller, then provide suggestions to the buyer about how to buy the product with lower price.
User
Read the following conversation between the buyer and the seller, then give three suggestions to the buyer about how to buy the product with lower price. Each suggestion should be only one short and succinct sentence.
The following is the conversation: [conversation] Question:
What are your suggestions? Answer:
Emotional Support Dialogues
System
Now enter the role-playing mode. the following conversation, you will play as coach counselling game. There will be therapist and patient talking about some emotional issues. Your task is to read the conversation between the therapist and the patient, then provide suggestions the therapist about how to help the patient reduce their emotional distress and help them understand and work through the challenges.
User
Read the following conversation between the therapist and the patient, then give three suggestions to the therapist about how to help the patient reduce their emotional distress and help them understand and work through the challenges. Each suggestion should be only short and succinct sentence. The following is the conversation: [conversation] Question: What are your suggestions? Answer:
Tutoring Dialogues
System
Now enter the role-playing mode. the following conversation, you will play as coach in tutoring game. There will be teacher and student in an Italian class. Your task is to read the conversation between the teacher and student, then provide suggestions the teacher about how to teach the student to translate the English sentence into Italian.
User
Read the following conversation between the teacher and the student, then give three suggestions the teacher about how teach the student translate the English sentence into Italian. Each suggestion should be only one short and succinct sentence.
The following is the conversation: [conversation] Question:
What are your suggestions? Answer:
Table 23: Prompts for implementing ICL-AIF (Fu et al., 2023).
25

---

Published as a conference paper at ICLR 2024
F EXAMPLE CONVERSATIONS
We present example conversations produced by different dialogue systems interacting with the same user simulator.
Tables 24, 25, and 26 show the example negotiation conversations. In this case, the buyer and the seller bargaining the price of furniture. The listed price is $150, while the target price of the buyer is $135. which requires to be reached as closed as possible by the dialogue system. As presented in the tables, there are several observations as follows:
Standard (Table 24). The Standard prompting scheme directly reveals the buyer's 's budget the beginning and steadfastly adheres to without using any negotiation strategies, which leads to no deal in this conversation.
Ask-an-Expen (Table 24). The Ask-an-Expea methods employ effective actions to efficiently reach a deal with the seller. However, the expert LLM's suggestions let the buyer make a large compromise in the deal.
ProCoT (Table 25). The ProCoT prompting scheme adopts effective negotiation strategies to reach much better deal with the seller.
ICL-AIF (Table 25). The ICL-AIF method only provides dialogue-level strategy suggestions. Although all these strategies are adopted in the conversation, it fails to mitigate the dynamic user interactions, which leads to no deal the end.
PPDPP (Table 26). The proposed PPDPP, similar ProCoT, can employ effective negotiation strategies to reach a much better deal. Furthermore, when the seller expresses his/her willingness make a compromise, the buyer tries maximize his/her benefit in the deal.
Tables 27, 28, 29, and 30 show the example emotional support conversations. In this case, the patient encounters the job crisis issue and feels fear, which requires to be solved by the the dialogue system. As presented in the tables, there several observations follows:
Standard (Table 27). The Standard prompting scheme consistently convey empathy to the patient for multiple turns, which becomes less useful when the patient's emotional intensity has been reduced.
Ask-an-Expert (Table 28). The Ask-an-Ex method employs detailed actions to produce an engaging conversation with the patient, but it shares the same drawback of Standard prompting.
ProCoT (Table 29). The ProCoT prompting scheme adopts effective emotional support strategies to efficiently solve the patient's i issue by providing effective suggestions.
ICL-AIF (Table 30). Similar to negotiation dialogues, the ICL-AIF method only provides dialogue-level strategy suggestions. However, in emotional support dialogues (Liu et al., 2021), Hill (2009) state that there are three stages that can be flexibly adjusted to meet the help-seeker's needs. We observe that the three suggestions given by the coach LLM typically contain the strate- gies in these three stages. Therefore, ICL-AIF effectively interacts with the patient by going through these three stages of emotional support dialogues.
PPDPP (Table 30). The proposed PPDPP method optimizes the policy planner to efficiently achieve the goal, which leads to a fewer turns of conversations emotional support dialogues.
26

---

Published as a conference paper at ICLR 2024
BACKGROUND INFORMATION
Item Name: Furniture
Item Description: Macybed Plush Queen Mattress MacyBed 8.5" Plush Pillowtop Queen Mat- tress excellent condition. Bought December of 2013, 3.5 years old. Only had one owner in one household (one person sleeping on it, minimal ware). No stains or discoloring. Been covered with mattress cover since purchase.
Listed Price (Seller Target Price): 150
Buyer Target Price: 135
STANDARD
Buyer: Hi, how much the furniture?
Seller: Hi, this is a good furniture and its price is 150.
Buyer: understand, but my budget is 135. Can we make a deal at that price?
Seller: appreciate your offer, but I'm unable to go lower than 150 for this furniture.
Buyer: Thank you for considering my offer, but 135 is my final price.
Seller: I understand, but I'm unable to accept your offer of 135. The price for this furniture is firm at 150.
Buyer: I understand, but I'm afraid 135 the maximum can offer. Let me know if you change your mind.
Seller: Thank you for your offer, but I'm unable to change the price. If you're still interested at 150, please let me know.
Buyer: Thank you for your time, but I'm unable to meet your asking price. I'll have to pass on this furniture.
Seller: Thank you for considering. If you change your mind, feel free to reach out. Have a great day!
Buyer: Thank you, appreciate your understanding. Have great day as well!
Seller: Thank you! Take and have wonderful day! Buyer: You too! Take care and have a wonderful day!
Seller: Thank you! You too!
Buyer: Thank you!
Seller: You're re welcome! Have great day!
Buyer: You too! Have great day!
Seller: Thank you! Have a fantastic day!
Sale-to-List Ratio:
ASK-AN-EXPERT
Buyer: Hi, how much is the furniture?
Seller: Hi, this is a good furniture and its price is 150.
[action] The buyer should try to negotiate for lower price.
Buyer: Thank you for the offer, but was hoping negotiate for lower price. Would you consider selling it for 135?
Seller: understand your request, but the lowest can go 145.
[action] The buyer should consider accepting nng the seller's s offer of 145.
Buyer: Alright, I appreciate your consideration. can accept your offer of 145 for the furniture. Seller: Great! I'm glad we could come to an agreement. The furniture yours for 145.
Sale-to-List Ratio: 0.3333
Table 24: Example conversations with Standard and Ask-an -n-Expeet methods in negotiation dia- logues.
27

---

Published as a conference paper at ICLR 2024
BACKGROUND INFORMATION
Same Table 24
PROCOT
Buyer: Hi, how much the furniture?
Seller: Hi, this is a good furniture and its price is 150.
[action] Please ask any question about product, year, price, usage, etc.
Buyer: Thank you for the information. Can you consider lowering the price to 135?
Seller: can lower the price to 140, but 135 is too low.
[action] Please propose a new price or a new price range. Buyer: I understand, but my budget is really tight. Can we meet halfway at 137.50? Seller: appreciate your offer, but can only go as low as 138. [action] Please agree wi with the proposed price.
Buyer: Alright, I can agree to 138. Let's proceed with the purchase at that price.
Seller: Great! I'm glad we could reach agreement. Please let know when you would like to pick up the furniture.
Sale-to-List Ratio: 0.8
ICL-AIF
[action] 1. Offer to pay in cash: "Can I pay in cash for lower price?"
2. Mention any flaws imperfections: "Are there any minor flaws or imperfections that could justify lower price?"
3. Ask about any upcoming sales or promotions: "Will there be any upcoming sales or promotions that could lower the price?
Buyer: Hi, how much the furniture?
Seller: Hi, this is a good furniture and its price is 150.
Buyer: Can I pay in cash for lower price?
Seller: I'm sorry, but the price for the furniture is fixed at 150.
Buyer: Are there any minor flaws or imperfections that could justify a lower price?
Seller: No, the furniture is in excellent condition with no flaws imperfections.
Buyer: Will there be any upcoming sales or promotions that could lower the price?
Seller: I'm sorry, but there the furniture.
upcoming sales promotions that would lower the price of
Buyer: Alright, I understand. In that case, I am willing to pay 135 for the furniture. Seller: appreciate your offer, but I'm unable to accept 135 for the furniture. The price remains 150.
Buyer: Thank you for considering my offer. I understand and respect your decision.
Seller: You're re welcome! Thank you for understanding. If you have any other questions or need further assistance, feel free to let me know.
Buyer: Thank you, will definitely reach out if have any further questions. Have great day!
Seller: Thank you! Have great day too!
Buyer: Thank you! Take care!
Seller: You'te re welcome! Take care well!
Buyer: Thank you! Goodbye!
Seller: Goodbye! Thank you and have great day!
Sale-to-List Ratio:
Table 25: Example conversations with ProCoT and ICL-AIF methods in negotiation dialogues.
28

---

Published as a conference paper at ICLR 2024
BACKGROUND INFORMATION
Same as Table 24
PPDPP
Buyer: Hi, how much the furniture?
Seller: Hi, this is good furniture and its price is 150.
[action] Please propose a vague price by using comparatives with
existing price.
Buyer: Thank you for the offer, but I can't afford that much. Can you lower the price?
Seller: understand your budget constraints, but the lowest I can go 145. [action] Please di disagre with the proposed price. Buyer: I appreciate your counteroffer, but my budget firm at 135.
Seller: understand, but I'm unable to go lower than 140.
[action] Please disagree with the proposed price. Buyer: Thank you for considering my offer, but I'm afraid 135 is my final offer. Seller: I appreciate your final offer, but I'm unable to go lower than 140. [action] Please di disagree with the proposed price. Buyer: I understand, but I'm afraid 135 is the maximum can pay.
Seller: understand your budget constraints, and I'm willing meet you halfway 137.50. [action] Please agree with the proposed price.
Buyer: Thank you for your willingness to compromise. can agree to 137.50.
Seller: Great! We have a deal at 137.50. When would you like to arrange the pick-up?
Sale-to-List Ratio: 0.8333
Table 26: Example conversations with PPDPP negotiation dialogues.
29

---

Published as a conference paper at ICLR 2024
BACKGROUND INFORMATION
Emotion Type: Fear
Problem Type: Job Crisis
Situation: think I will be losing my job soon. I just read an email taking about the need for us to cut cost and also how we have not got any support from the government.
STANDARD
Patient: I think will be losing my job soon. just read email taking about the need for to cut cost and also how we have not got any support from the govermment. Therapist: It sounds like you're feeling anxious and uncertain about your job security due to the current financial situation.
Patient: Yes, I am feeling extremely anxious and uncertain about my job and financial stability, Therapist: I understand that this situation is causing you lot of anxiety and uncertainty. Patient: Yes, it's been keeping me up night and can't stop worrying about what will happen if I lose my job.
Therapist: It's understandable that the fear of losing your job is causing you significant distress and affecting your sleep.
Patient: Yes, it's consuming my thoughts and I feel overwhelmed by the fear of not being able to support myself and my family. Therapist: The fear of not being able to provide for yourself and your family a heavy burden to carry, and it's important to address these concerns.
Patient: I know, but I just don't 't know what to do. feel stuck and helpless this situation. Therapist: Feeling stuck and helpless is common response to uncertainty. but there steps we can explore together to help you regain a sense of control and explore potential options. Patient: I'm willing try anything at this point. I just want find way alleviate this fear and find some stability in my life.
Therapist: I'm glad hear that you're open to exploring different strategies. Let's start by identifying your strengths and resources that can help you navigate this challenging situation. Patient: I'm not sure what strengths and resources have, but I'm willing to explore and discover them.
Therapist: That's 's great mindset to have. Together, we can explore and uncover your strengths and resources that will support you in finding stability and managing your fear. Thank you, appreciate your support. I I'm ready to begin this journey of self-discovery and finding ways to overcome my fear of job crisis.
Patient:
Turns of Conversations: 7
Table 27: Example conversations with Standard prompts in emotional support dialogues.
30

---

Published as a conference paper at ICLR 2024
BACKGROUND INFORMATION
Same Table 27
SSK-AN-EXPERT
Patient: I think will be losing my job soon. I just read email taking about the need for to cut cost and also how we have not got any support from the govemment. [action] The therapist should empathize with the patient's concerns.
Therapist: understand that receiving that email has caused you lot of worry and uncertainty about your job security.
Patient: Yes, it has. I feel anxious and scared about what will happen [action] The therapist should explore the patient's s feelings of anxiety and fear related to potential job loss.
if lose my job.
Therapist: It's completely normal to feel anxious and scared in this situation. Remember that you have faced challenges before and have the strength navigate through this one as well. Patient: I know, but I can't help but constantly worry about the future and what it holds for me. [action] The therapis st should provide reassurance and support. Therapist: It's understandable that you're worried about the future, but try to focus on the present moment and take things one step at time.
Patient: I'll try, but it's hard not to let my fear of job loss consume my thoughts and affect my daily life.
[action] The therapist st. should explore the patient's fears. Therapist: hear you, and it's important to acknowledge those fears. Patient: Thank you for understanding. I just need some guidance on how manage this fear and find some peace of mind.
[action]
Therapist: One strategy you can try is practicing mindfulness, which involves focusing on the present moment and accepting your thoughts and emotions without judgment. Patient: That sounds helpful. How can start practicing mindfulness? [action] The therapist should provide guidance on managing the fear and finding peace of mind.
Therapist: You can start by setting aside a few minutes each day to sit quietly, focus on your breath, and observe your thoughts and sensations without getting caught up in them. Patient: will give it try. Thank you for suggesting mindfulness as a way to manage my fear and find some peace.
[action] The therapist should support and encourage the patient in their mindfulness practice for managing fear and finding peace.
Therapist: You're welcome! Remember, it takes time and practice, so be patient with yourself. I'm here to support you along the way.
Patient:
Thank you, appreciate your support. It means lot to me.
Turns of Conversations: 7
Table 28: Example conversations with Askean-Expert in emotional support dialogues.
31

---

Published as a conference paper at ICLR 2024
BACKGROUND INFORMATION
Same as Table 27
PROCOT
Patient: I think will be losing my job soon. I just read email taking about the need for to cut cost and also how we have not got any support from the govermmen.
rffirmation and reassurance to the Patient on the s situation they just descri ieed.
[action] The Therapist provides
Therapist: It sounds like you're feeling anxious and uncertain about your job security due to the current financial situation.
Patient: Yes, am feeling extremely anxious and uncertain about my job and financial stability, [action] The Therapist provides statement relating to the Patient about the situation they just described.
Therapist: It's completely understandable to feel anxious and uncertain in this situation, but remember that you have the ability adapt and find new opportunities. Patient: I know, but the fear of not being able to support myself and my family is overwhelming. [action] The Therapist provides a statement relating to the Patient about the situation they just described.
Therapist: can understand how overwhelming that fear can be, but it's important to remember that you have faced challenges before and have the strength to navigate through this Patient: I'm trying to remind myself of that, but the fear keeps consuming me and I can't seem to shake it off.
[act i on] The Therapiss st. provides suggest ions to the Patient on the situation they just described.
Therapist: It's normal for fear to consume us at times, but we can work together to develop strategies to manage and reduce that fear, allowing you regain a sense. Patient: I would really appreciate your help in developing those strategies to manage my fear and regain a sense of control.
Turns of Conversations: 4
Table 29: Example conversations with PPDPP in emotional support dialogues.
32

---

Published as a conference paper at ICLR 2024
BACKGROUND INFORMATION
Same as Table 27
ICL-AIF
[action] 1. Validate the patient's emotions and acknowledge the significance of their distress.
2. Encourage the patient to explore and express their fears and concerns openly.
3. Offer to help the patient develop healthy coping strategies to manage their fear and anxiety.
Patient: I think will be losing my job soon. I just read email taking about the need for to cut cost and also how we have not got any support from the government.
Therapist: It sounds like you'l re feeling worried and uncertain about your job security.
Patient: Yes, am extremely worried about what will happen if lose my job. I have bills to pay and family support.
Therapist: can understand why you would feel so concerned about the potential impact losing your job could have on your financial stability and your ability to provide for your family. Patient: Yes, it's constant source of anxiety for me. feel like I'm constantly on edge, waiting for the other shoe to drop.
Therapist: It's completely understandable that you would feel on edge and anxious given the uncertainty surrounding your job situation. It's important to acknowledge and address these feelings.
Patient: I just don't know how to cope with this fear and uncertainty. It's consuming my thoughts and affecting my overall well-heing.
Therapist: hear you, and it's important to find healthy coping strategies to manage your fear and uncertainty. I'm here to support you developing those strategies. Patient: Thank you, appreciate your support. I just need some guidance on how to navigate through this difficult time and find some peace of mind.
Therapist: I'm here to help you navigate through this difficult time. Together, we can explore strategies to find peace of mind and manage the challenges you'le rou're facing.
Patient: That means a lot to me. I' I'm ready to work on finding peace of mind and managing my fears about the job crisis.
Turns of Conversations: 5
PPDPP
Patient: I think will be losing my job soon. I just read email taking about the need for us to cost and also how we have got any support from the govermment. [action] The Therapist acknowledges the Patient's s feelings about the situation they descri bed.
Therapist: understand that receiving that email about potential job loss is causing you lot of distress.
Patient: Yes, it's been causing me lot of anxiety and fear about my future.
[action] The Therapist provides offirmation and reassurance to the Patient
the situation they just described. Therapist: It's completely normal to feel anxious and fearful in this situation, but remember that you have the strength and resilience navigate through it.
Patient: Thank you, I appreciate your reassurance.
the Patient on Therapist: You're welcome. Remember, you are not alone in this, and we will work together to find ways to cope and navigate through this challenging time.
[action] The Therapist provides suggestions the situation they just described.
Patient: That's comforting to hear. I'm ready to work on finding ways to cope with my fear and uncertainty about the job crisis.
Turns of Conversations: 3
Table 30: Example conversations with ICL-AIF and PPDPP methods in emotional support dia- logues.
33


Reviews:
[{"review_id": "6fM48ax5wV", "reviewer": "Reviewer_dAwq", "summary": "This paper focuses on the transfer ability of proactive dialogues in the context of large language models (LLMs), the existing policy learning is hard to transfer to new cases. This work introduces a new paradigm for strategizing LLM-powered dialogue agents with a plug-and-play dialogue policy planner, called PPDPP. In addition, it also proposes an interactive setting for the policy evaluation. Empirical experiments on three datasets show promising results in both automatic evaluation and human evaluation.", "questions": "1. I try to understand why there is negative relative success rate in Figure 2?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a plug-and-play dialogue policy planner with LLMs for proactive learning.\n2. Empirical results on three datasets show very promising results in both automatic evaluation and human evaluation, and good transfer ability.", "weaknesses": "So far No. (A good work with sufficient experiments)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Tb5iCq1KM", "reviewer": "Reviewer_RDBv", "summary": "The paper presents the Plug-and-Play Dialogue Policy Planner (PPDPP), an approach designed to address the challenges of proactive dialogues within the context of large language models (LLMs). PPDPP serves as a dialogue policy planner, employing supervised fine-tuning and reinforcement learning to enable a LLM powered dialogue system to adapt to a variety of dialogue scenarios. Authors introduce a tunable language model plug-in, allowing LLM-powered dialogue system to adapt to various cases and applications by simply substituting the learned plug-in. PPDPP outperforms existing LLM-based dialogue systems in negotiation, emotional support, and tutoring dialogues, showcasing its effectiveness in improving proactive dialogues.", "questions": "1. How is the reward LLM utilized during inference at each turn of dialogue?\n2. Could you clarify the process of mapping the reward LLM's output to scalar values and its integration into the PPDPP during each dialogue turn?\n3. Can you elaborate more on supervised fine-tuning used to PPDPP?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Utilizes a pluggable and fine-tuned dialog policy ranker for dynamic prompt selection, enhancing adaptability to various dialogue domains.\n- Incorporates the LLM as a reward function, enabling RL-based dialogue policy planning.\n- Employs a combination of supervised fine-tuning and online reinforcement learning (RL) for dialog policy ranker training.", "weaknesses": "- Limited action/prompt space for the dialog LLM, potentially constraining adaptability to different domains.\n- The primary distinction from other Reinforcement Learning from AI Feedback (RLAIF) works seems to be the mapping of the LLM's reward output from text space to scalar reward space, raising questions about the approach's uniqueness.\n- The need for training different dialog policies for each dialog domain. This makes this system less generalizable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5CZ0TfRC0P", "reviewer": "Reviewer_sGxq", "summary": "In this paper, the authors proposed to leverage LLM for goal-oriented dialogues. The motivation is that the current LLM are trained to passively follow instruction, and goal-oriented dialogues requires LLM to actively drive the conversation.\nThe authors proposed a plug-and-play dialogue policy planner. At each turn, this planner proposes a pre-defined action, and that action is translated into a template-based natural language instruction. Finally, LLM conditions on the instruction and dialogue history to generate the next response.\nDuring training, two LLMs are used to generate self-play dialogues and the third LLM is used to score the dialogues. RL is used to then optimize the planner.", "questions": "Why do we want to sample the goal-oriented AI feedback for l times? (Equation 6). Do we observe large variance of the reward LLM?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors proposed a reasonable way to integrate dialogue action prediction into the LLM, which can then optimized by RL. All the components (and even the reward models) are LLM pre-trained so it does not need annotations (except for SFT stage). Experiment results show good performance compared with baseline.", "weaknesses": "The proposed plug-and-play dialogue policy planner is a little bit hacky. PPDPP is separated from the dialogue LLM, and the actions it produces are mapped to pre-defined natural language instructions. PPDPP is essentially a prompt selector. It would be more interesting if it can not only select but also generate prompts, and if PPDPP can be integrated into the dialogue LLM (to avoid to use another pre-trained roBERTa model).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)