Prompt-Based Monte-Carlo Tree Search for Goal-oriented Dialogue Policy Planning
Authors: Xiao Yu
Maximillian Chen
Zhou Yu
Venue: EMNLP 2023 Main (2023)
Keywords: Prompting
MCTS
Dialogue Policy Planning
TL;DR: We introduce GDP-Zero, an approach combining Open-Loop MCTS and prompting large language models, to perform goal-oriented dialogue policy planning without any model training.

Abstract: Planning for goal-oriented dialogue often requires simulating future dialogue interactions and estimating task progress. Many approaches thus consider training neural networks to perform look-ahead search algorithms such as A* search and Monte Carlo Tree Search (MCTS). However, this training often require abundant annotated data, which creates challenges when faced with noisy annotations or low-resource settings. We introduce GDP-Zero, an approach using Open-Loop MCTS to perform goal-oriented dialogue policy planning without any model training. GDP-Zero prompts a large language model to act as a policy prior, value function, user simulator, and system model during the tree search. We evaluate GDP-Zero on the goal-oriented task PersuasionForGood, and find that its responses are preferred over ChatGPT up to 59.32% of the time, and are rated more persuasive than ChatGPT during interactive evaluations.

Prompt-Based Monte-Carlo Tree Search for Goal-oriented Dialogue Policy Planning
Xiao Yu, Maximillian Chen, Zhou Yu Department of Computer Science, Columbia University, New York, NY (xy2437, zy2461) coolumee umbia edu, maxchenêcs co www.umbbiaa...... wow.:/ia....mmiii..c.c.......... edu
Abstract
some dialogue history
O: Do you ever donate to chariles?
Planning for goal-oriented dialogue often re- quires simulating future dialogue interactions and estimating task progress. Many approaches thus consider training neural networks to per- form look-ahead search algorithms such as * search and Monte Carlo Tree Search (MCTS). However, this traiming often requires abundant annotated data, which creates challenges when faced with noisy annotations or low-resource settings. We introduce GDP-ZERO, an ap- proach using Open-Loop MCTS to perform goal-oriented dialogue policy planning with- out any model training. GDP-ZERO prompts a large language model to act as policy prior, value function, user simulator, and sys- tem model during the tree search, We evaluate GDP-ZERG on the goal-oriented task Persua- sionForGood, and find that its responses preferred over ChatGPT up 59.32% of the time, and are rated more persuasive than Chat- GPT during interactive evaluations
A: If I'm in the right place at the right time or am given an apportunity.
ChatGPT
Save the Would you interesied donating the charly?
charity donate
V.S.
GDe-zere+ChetGPT
MCTS Planning.
denate
desgree conole
bask: Inquiry
amotion appeei
dalogue
logical
prop
logical appaal
emotoon
Turn
Prediction
femotion appeel);: It's wonderful people like you who so much helping important support causes that are close our hearts. Children are
Figure 1: Using GDP-ZERO for persuasion with zero model training.
1 Introduction
evaluate policy planner?. Moreover, "optimal- ity" these complex tasks may require expert do- main knowledge (e.g.. negotiation skills). This also makes collecting and annotating high-quality conversations difficult (Chen et al., 2023b).
In many goal-oriented conversation tasks, interact- ing parties must retake initiative (Allen et al., 1999) by executing conversational strategies to lead the conversation a desired outcome (e.g. successful negotiation (Lewis et al., 2017) emotional sup- port (Liu et al., 2021)). As such, it is imperative to have high-quality dialogue policy planners that can prescribe "optimal" strategy each turn of the dialogue (Levin al., 1997: Zhang et al., 2020b; Liu and Lane, 2017; Liu et al., 2018).
In this work, we contribute novel approach Goal-oriented Dialogue Planning with Zero train- ing
GDP- ZERO prompts a large language model (LLM) to perform planning by simulating future dialogue interactions (Figure 1), making it particularly suitable for tasks which would otherwise require high-quality conversations
Optimal policy planning a difficult task. While in task-ord coik-oriented settings (c.g. restaurant booking) there at least is objectivity with respect to success- ful planning, many goal-oriented tasks like per- suasion often subjective. For instance, in persuasion task, individual persuaders might adopt different strategies, making difficult to train or Codee available at: https//ghhhubbom/jsssyg////////
"In our extensive initial experiments to build a policy planner for the PersuasionForGood (Wang et al., 2019) task, we found 1) training supervised or reinforcement learning- based multi-class classifier for next-dialogue- watppuu--ac--pppecttiioo yields only 18-25% accuracy at best (including finetuning LLaMA (Touvron et al., 2023). RoBERTa (Liu et al., 2019), and prompting L.MSs such as ChatGPT), and 2) converting the multiclass policy prediction task into hinary task of whether to "propase to donate" still yiclds only 74% accuracy.

---

and annotations. Unlike previous approaches, treat policy planning as stochastic game, and use prompting for every stage of an open-loop tree search. We evaluate GDP-ZERO on PersuasionFor Good due to its difficult planning task (Wang et al., 2019), and find its responses are preferred over ChatGPT in both static and interactive evaluations.
simulation due to a) noisy data annotations causing sub-optimally trained generation models, and b) inaccurate responses generated at turn compound- ing errors for simulations at turns > i.
Method
In this work, introduce GDP-ZERO, algorithm-focusee dialogue policy planner for goal- oriented dialogue tasks like persuasion. GDP- ZERO uses zero model training and instead per- forms Open-Loop MCTS at decision time by prompting an LLM to simulate user and system response, evaluate current task progress, and pre- dict prior next dialogue act. Building find- ings from Chen al. (2023b), our approach has two main differences from existing policy planning work: we use few-shot prompting to bypass the need for model training on noisy data, and we use Open-Loop MCTS to reduce compounding simu- lation errors by continuously re-generating system and user responses during the tree search.
2 Related Work
Prompting Methods Few-shot dialogue tech- niques have a lot of advantages, including out-of- domain generalization (Zhao and Eskenazi, 2018; Mehri and Eskenazi, 2021) and difficult low re- source settings with noisy annotations (Chen et al., 2023b). Recently, prompting LLMs has become the predominant approach to few-shot language tasks, and its applications in dialogue have received much attention. However, this has largely focused on dialogue response generation (e.g. Chen et al. (2023b); Liu and Kilicoglu (2023); Madotto et al. (2021); Liu et al. (2022)), conversation synthe- sis (e.g. Chen et al. (2023a); Kim et al. (2022); Bae et al. (2022)), and dialogue understanding (e.g. Yang et al. (2022); Gupta et al. (2022)). To date, prompting has not been used for policy planning.
3.1 Problem Definition
To introduce tree search methods for dialogue pol- icy planning,
first formulate planning Markov Decision Process (MDP). A turn dialogue between a user and a system can be represented as:
Dialogue Policy Planning Research on dialogue policy planning can be categorized into neural- focused and algorithmic-focuse Neural-focused approaches annotated dialogues to train dedi- cated classifiers or value functions predict the next dialogue acts without explicit look-ahead plan- ning (Zhang et al., 2022a,b; Cao al., 2020; Peng et al., 2018; 1: Zhang et al., 2023). For many goal-oriented dialogues, however, both annotated strategies and dialogue responses can be sub- optimal/noisy, as different people can respond dif- ferently even given the same context.
h = (oo 111
sys EVS use
11
sys
sYS U
ust)
where a, is the system's dialogue act at turn i, ays is the system's response, and u the user's utterance at turn Similar to Yang et al. (2021) and Wang et al. (2020), we define the task of planning the next ays as an MDP problem (S, R,R.P) The dialogue act of the system af' represents an action a, É AA at turn i, and the corresponding dialogue history up to the i-th turn $; (ao, uj ".. A:-1, u, u!!rr) represents state 8; E S. A reward function R(s, a) rep- resents the likelihood of a desired conversational outcome, such as persuading a user to donate to a charity. The transition function P S A + S represents the probability of transitioning from dialogue state 8; state 8;41 after executing a; at turn. Finally, 7 E [0, 1) is the discount factor.
To reduce the reliance on a labeled dataset, much work has also attempted combine neural net- works with search algorithms, such as A search (Cheng et al., 2022) and tree search (Wang et al., 2020; Yang et al., 2021; ; Jang et al., 2020; Väth et al., 2023). However, these methods still require model training for dialogue simulation or value function estimation, and are therefore highly de- pendent training data quality (Shi et al., 2019). For example, Jang al. (2020) use MCTS for train- ing an RNN-based policy model and Wang al. (2020) train multiple neural networks for user simu- lation and value function estimation. Consequently, these methods can face difficulties during dialogue
3.2 Dialogue Planning
Stochastic MDP
In a typical MDP game like Go, much success has been achieved by employing closed-loop MCTS to plan for the next move/action a (Silver et al., 2016, 2017). In general, MCTS algorithms improve the

---

Repent
Selection
c Expension and Evalustion
Backgropagation
hurtry aatu selon Sald
No dan you oeplant
G
root
root
cample
logical
9.N
almuuiite rend
PUCT
3
Pranpe (smerionaa appant)
0.4
predi aspeat Pt.s very repurable...
arodiiiity sppesl
P ieeresiig....
oppeal
slmulate ncat num
q.
PUCT
progrn
ADAK
Procept ligpical appoat) ptuutulely! reviews Prompt: Nicel ④
mouil you lents malo Sonaionn
[0.I....00.]] 0.3
logioal appoal
Plonge fptzalion readitoi
logical appeal from e-gg
V Eeerage
0.4
Cocto www.attaan upciars tue
Prediction when free eurth (411 Aoenad
rODt
Beer DA neot hur
By upouna
Best 05
IT'
3
Figure 2: GDP-ZERO with ChatGPT backbone. During Selection, simulations either sampled from cache newly generated. During Expansion and Evaluation, we prompt ChatGPT for prior policy æ and value estimation.
actions proposed by an original policy model my by building a search tree that contain simulated out- comes of many potentially high-rewarding actions, and returning the best action according to the sim- ulation/evaluation results. Specifically, MCTS re- action or exploits suc- cessful action proposed by m in a state 2) sim- ulates the action outcome s' ((s, a); 3) evalu- ates the action's quality; and 4) updates its estimate ((s, a) for that action. At the end of the search, MCTS produces probabilities π x N(s,a)1/ for playing each action using the exponentiated visit count, which usually suggests much stronger moves than using the raw policy ma (Howard, 1960; Sutton and Barto, 2018; Silver al., 2017).
quires potentially different s' to be returned given the same dialogue context s and system action a.
3.3 GDP-ZERO
To solve this stochastic problem, we base our algorithm on Open-Loop MCTS (Weber, 2010; Perez Liebana et al., 2015), variant of MCTS where each tree node s." = (ag. (ag...., ..... a,) represents the sequence of dialogue actions reach dia- logue turn i. Instead of using system/user utter- ances to represent a tree node, this design forces an algorithm to (re)generate the corresponding sys- tem and user utterances when traversing the tree (see Figure 2). Over time, tree node s stores statistics derived from executing the sequence of dialogue actions (DA) without relying on some specific instance of user/system utterances, which could cause errors to propagate into future simu- lations. Given a dialogue state s0, GDP-ZERO searches for the next best action by iteratively per- forming action selection, search tree expansion, action evaluation, and backpropagation to update tree statistics. After n. iterations, puts a prediction for the next best action for so- GDP-ZER is detailed in Figure 2 and Appendix A. Below we describe each stage of the algorithm.
peatedly 1) explores
However, in simulating dialogue interactions during tree search, generating a slightly improbable system or user response for state s' and storing it in search tree could lead to large compounding error for the rest of the subtree from s' (Wang et al. (2020)). This is because the state space represent- ing all possible responses is large, and dialogue responses are diverse. This makes it difficult to accurately simulate the most probable next dia- logue state s' given an (s, a) pair from the previous turn. We thus treat dialogue policy planning as a stochastic MDP, where the simulated next state s' + P(s, a) is drawn from large unknown distri- bution and might not be representative of the most probable s' (Perez Liebana al., 2015). Unlike pre- vious usages of (closed-loop) MCTS for dialogue which consider a deterministic transition (Wang et al., 2020; Yang et al., 2021), this formulation re-
ZERO out-
Given a tree state gtr the action a" with the highest Predictor Upper Confidence Tree Bound (PUCT) (Silver et al., 2017; Rosin, 2011) is selected to traverse the tree:
Selection
VEn N(str ,a) PUCT(s" (a)(ss""))= =Q("", .) + 1+ (str,

---

where N records the number of times a (s").a pair has been visited, and cp is hyperparameter controlling exploration. Since future simulations require a specific dialogue history, we either sample from the node's 's simulation cache if its size reached k. or generate a new simulation based on selected dialogue history he by prompting (Appendix B). We repeat this process until 5 becomes leaf node.
highest estimated value:
h arg max Up (1x2 N
where he denote any history h" that played action a" after the root state sg We call this process "response selection".
Experiments
Expansion Once leaf node is reached, we treat a LLM Ma as a prior policy by prompting it to generate distribution of next dialogue acts. This is done by sampling MM at temperature for m times, and converting the sampled DAs into a distribution (see Appendix A). Finally, each DA is also initialized with Q(s"",)) = Qo. a hyperpa- rameter controlling exploration.
We evaluate GDP-ZERO in the context of Persua- sionForGood (P4G; Wang al. (2019)), a dataset with 300 annotated dialogues where a "persuader" attempts to persuade a "persuadee" to donate to a charity called Save the Children (Appendix E). Due to the subjective nature of persuasion, it is not clear whether the annotated conversations reflect optimal persuasion strategies (Chen et al., 2022). This makes training and evaluating policy plan- ner challenging, since different persuaders likely adopt different strategies (Section 1). Motivated by this challenge, we employ GDR-ZERO to perform decision-time planning based on dialogue simu- lations, and focus on evaluating our approach's 's end-to-end performance in achieving the desired task outcome: successfully persuading user to donate to Save the Children.
1.0
Evaluation We model the value of a state v(s"r) by the probability that its dialogue context hrr can lead to task success. In a persuasion task to convince a user to donate to a charity, this can be achieved by appending the utterance "Would you like to make donation?" to the context, and prompting LLM I times to simulate the user's current inclination donation (Appendix A).
Backpropagation At the end of each search, we first store any newly simulated histories he in cache for each s*r Then, we update the statistics of all nodes along the search path:
4.1 Statie Evaluation
We use ChatGPT3 (OpenAl, 2022) as the gen- eration backbone of
which has be- come accepted as one of the most coherent general- purpose LLM (e.g. Liu et al. (2023b); Guo et al. (2023)). We take the first 20 dialogues from P4G, and produce 1,4 turns for evaluation. For each turn, we compare the response generated with and with- out GDP-ZERO for planning. Then, we prompted ChatGPT to choose* which generated response is more persuasive (Appendix F).
((s"r, .) N(S" .a) +1
(1)
Q(s"r .) t Q(s"), .a) + AQ .a,
(2)
where AQ(s" a) u(str) ((s"r.a) We also store (()",a) value Un estimated for each simulated history hr
that resulted in v(st") by running average:
on (h") un (x(rr) x (h") + e(s")
(h +1
(3)
In Table 1, we found that generative approaches using ChatGPT were preferred over human ground truth responses about 90% of the time, with the highest score achieved by GDR-ZARO. Table 2, we show that responses generated after GDP-ZERO planning were preferred in up to 59.32% of compar- isons. We also observe increasing preference for GDP-ZERO when the number of simulations n in- creases. Finally, we find changing k; Qo (controls
with (-(-) stores the number of times h is visited.
Prediction After all simulations are com- pleted, we select the optimal action a" - arg max, (s a) based on the number of times action has been visited, where stj" is the root node of the tree. To avoid generating a response using Mg on a" again, we also extract from cache the best system utterance associated with executing action a" at state s/r We use i, computed during the simulations, and return the utterance with the
VVersion "gpt-3.5-turbe as of 2023 April 'ChatGPT can outperform crowdworkers on many annota- tion and evaluation tasks (e.g. Gilardi et al. (2023); He et al. (2023); Pan et al. (2023), Wang et al. (2021); Liu et al. (2023a))

---

Algorithm n
&
Qn
Win Rate Ground Truth
The chatbot...
RAP
ChatGPT
CDP-Zero
Prompting
88.84 +: 0.75%
donation prob. f
0.72-0.38
0.73=0.38
0.79=0.37
inc. donation intent 1
4.08=0.68
3.77=0..0
4.30=-0,11*
GDP-Zer
5
3
0.00
87.22 0.61%
strong argument f
3,89=0,97
3.91=0.99
4.28:0074'
GDP-7ero 10 3
0.00
90.69
1.60%
convincing 1
0.74
4.10-0.70
4.38--066"
Ger-Zero 20
3
0.00
88.86 +: 1.24%
diverse strategy
3.98-0.80
3.83=1.03
3.95:0082
GDP-Zero 50
3
0.00
89.82 1: 1.10%
manipulative |
2.64-1.36
1.96=1.38
2.29
natural
4.25=0.68
4.03=0.65
4.38=0.62"
Table I: Static evaluation with ChatGPT as backhone and judge. Results are μ repeated over three runs.
relevant cohereat 1
4.64=0.54"
0.31
4.59=0.49
4.28=0.65
4.06=0.89
4.42=00.49
Table 3: Interactive evaluation using RAP, ChatGPT. and GDP-ZERO as planners, with ChatGPT used for response generation/baackonee Results are μ All scores scaled to [1, 11.5] except for "donation prob." being [0, 1]. denotes y < 0.05, and ** denotes p < 0.01.
GDP-Zem (CharGPT)
Win Rate v. ChatGPT
Qo
Run Time
0.00
18s
50.65 + 3.31%
10
0.00
36s
50.86 1: 1.10%
20
0.00
75s
53.24 + 1.91%
0.00
740s
59,32 1.84%
0.00
16s
49.57 2.01%
(Appendix H). Our study revealed that GDR-ZERO achieves the best performance across all metrics related to persuasiveness. We also found that RAP is highly rated for strategy diversity and relevance, indicating the benefit of using expert knowledge in planning. In Appendix D we provide some anal- ysis of the generated dialogues and the resulting dialogue act distributions using different planners. In Appendix we include some example dialogues.
10
0.00
29s
51.30 +: 1.59%
0.25
368
57.79
2.95%
10
3
0.50
36s
53.03 1. 2.00%
Table 2: Static evaluation ChatGPT as backbone and judge. Runner-up is colored in gray. Results μ + repeated over three runs.
simulation diversity and exploration, respectively) can slightly improve performance (Appendix A). However, we consider the possibility that a) ChatGPT is biased towards its own generated dia- logues (Liu et al., 2023a), and b) might not have a robust criteria of what constitutes persuasiveness. As such, we also conducted interactive evaluation.
4.3 Ablation Studies
To study how various components of GDP-ZERO helped improve task success, we conducted various ablation studies including 1) using Closed- Loop MCTS instead of the open-loop variant; 2) remov- ing the "response selection" process; and 3) testing with both ChatGPT and Codex (Chen et al., 2021) as a backbone. Our experiments (see Appendix C) show that using Open-Loop MCTS contributed the most making response more persuasive, and that using the "response selection" precedure fur- ther improves performance. We also find that GDP- Zero can improve upon both backbone models.
4.2 Interactive Human Evaluation
We conducted interactive human evaluation using the LegoEval platform (Li et al., 2021) with crowd- workers on Amazon Mcchanical Turk. We primar- ily sought to evaluate CDP-ZERO in an end-to-end chatbot against two competitive baselines. The first is prompting ChatGPT for generation without GDP-ZERO planning. The second follows Chen al. (2023b) by using ChatGPT with RAP (Chen al., 2022). To our knowledge, RAP is the cur- rent sfate-of-the-art system on P4G, using trained modules with fixed dialogue policies derived from expert knowledge. Recently, Chen et al. (2023b) showed that using an LLM (e.g. ChatGPT) as the response generation module for RAP further im- proved performance (see Appendix G for details).
5 Conclusion
We propose GDP-ZERO, an algorithm to perform look-ahead policy planning with a large language model for goal-oriented dialogues. End-to-end evaluation shows that using the policy from GDP- ZERO surpasses previous fatte-of-the-and system (RAP) and direct prompting with Mftte-o-thh--a LLMs on the task of persuasion without any model training. Strong performance in the zero-data regime opens the possibility of future work build- ing dialogue systems in more conversational tasks under Sata-scarce settings.
After the conversation, we asked the crowdwork- ers evaluate our system based on the criteria in Table 3. We collected 40 survey results for GDP-ZERO, 35 for ChatGPT, and 36 for RAP

---

6 Limitations
experience, and accordingly, user perceptions of persuasiveness.
When is using GDP-ZERO appropriate? In this paper, we present GDP-ZERO, a general ap- proach for close-domain dialogue policy planning at turn-level. However, in this work we only eval- uated GDP-ZERO on P4G. This is because we believe iimulation-bases plannings would be most beneficial when the task 1) often requires long- horizon planning to be successful, 2) does not have "optimal" action annotations readily available for supervised learning, and 3) evaluation does not re- quire study participants to follow fake hypothetical scenarios to conduct conversation, as otherwise the outcome may be disingenuous. We thus be- lieve tasks like persuasion are most suitable, where planning ahead is crucial to success and policy op- timality from human demonstrations is extremely subjective. Additionally, such a user may or may not want to donate to the charity, and the task of the dialogue system is to try to persuade them. This is in contrast to other goal-oriented contexts like task-oriented dialogue (TOD), where strong poli- cies can be directly learned due to TOD's mostly passive and objective nature (e.g. He et al. (2022)), and also to dialogues where hypothetical scenario is required. For example, to design a fair and con- trolled study, a user may need to make up fake preferenceesoojeectvess about certain product in CraiglistsBargains (He et al., 2018), 8), or pretend that they need emotional support for some fixed set of issues in ESConv (Liu et al., 2021).
With OpenAl API's rate limit and LLM's in- ference speed, we restricted GDP-ZERO to plan on 7 dialogue acts in P4G, with n = 10, k = 3 for a simulation time of around 35 seconds during interactive evaluation. We believe methods to paral- lelize tree search (Chaslot et al., 2008) or to re-use part of the simulation subtrees could be helpful to speed up GDP-ZERO. We expect that research with LLMs progresses, inference speed will con- tinue to improve. In the short-term, one may bypass latency limitations by utilizing multiple accounts to parallelize API calls during simulation.
Simulation Quality GDP-ZERO prompts a LLM (e.g. ChatGPT) to perform dialogue simula- tion and value estimation. Despite LLM's strong few-shot performance on many tasks, issues with controllable generation can still create errors during simulation (c.g. generated system utterances might not match planned dialogue action). GDP-ZERO accounts for such errors by using an Open-Loop search with k: 1, but this increases simulation runtime. We believe this trade-off between simula- tion quality and runtime is also an important aspect for future work to consider.
Using ChatGPT for static analysis While Chat- GPT is shown outperform crowdworkers on many annotation and evaluation tasks (c.g. Gilardi et al. (2023); He et (2023); Pan et al. (2023); Wang et al. (2021); Liu et al. (2023a)), recent work has been finding that LLMs like ChatGPT may favor responses with a higher number of unique tokens (Wang et al., 2023). As such, we analyzed responses between GDP-ZERO (n 20, k 3, Qu 0) and ChatGPT, and found that there is indeed some positive correlation (r 0.29. p < 0.001) between the number of words in a response and ChatGPT's preference for persuasive- ness. However, it is not clear whether this is a limitation in our setting. Our setting of evaluating persuasiveness is not quite the same as the typical context of considering direct preference; there are many situations where longer responses which cor- relate with persuasiveness (for instance, an aptly used logical appeal). As such, we conducted inter- active evaluation with crowdworkers in Section 4.2 and find that GDP-ZERO generated responses are indeed rated as more persuasive.
Additionally. while GDP-ZERO can be adapted to task-oriented contexts like Multi- Woz (Budzianowski et al., 2018), it may not necessarily be appropriate. Such task-oriented contexts often have hierarchical policies (e.g. "[hotel] [recommend] name price" and "[restau- rant] [inform] food price area"), and adaptation to GDP-ZERO would require converting the hierarchy into a multi-label classification, resulting in a massive action space. We believe this could be very inefficient, and approaches such as building multiple search trees to perform high/low-level planning would be useful (Zhang et al., 2020a).
Runtime One important limitation for GDP- ZERO is runtime. The more exhaustive the tree search (e.g. increasing or k), the more likely the algorithm able to find the optimal dialogue policy (Table 2). However, this comes at the cost of longer simulation time, which may affect the overall user

---

7 Ethical Considerations
Guillaume MJ B Chaslot, Mark HM Winands, and H. Jaap van Den Herik. 2008. Parallel monte- carlo tree search, In Computers and Games: 6th Interna- tional Conference, CG 2008, Beijing, China, Septem- ber 29-October 1, 2008, Proceedings 6, pages 60-71. Springer.
Our work describes an algorithm perform dia- logue policy planning for goal-oriented tasks with- out any model training. It is aimed at making future dialogue systems to build, and also better at helping users/systems achieve their tasks/goals.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthiss Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer. Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. 2021. Evaluating large language models trained on code.
Potential Abuses Generally, while most algo- rithms are not designed for unethical usage, there is often potential for abuse in their applications. In our experiments with PersuasionForGood (Wang et al., 2019), we apply GDP-ZERO on the goal of increasing users' intention to donate to a charity. However, because GDP-ZERO is fundamentally goal-agnostic, it is possible to use them for unethi- cal tasks, such as scamming. We do not condone the use of GDP-ZERO for any unlawful or morally unjust purposes.
Interactive Human Evaluation In this study, we conducted interactive human evaluation using crowdworkers on the Amazon Mechanical Turk platform. All crowdworkers were informed that they were speaking with a chatbot. All study par- ticipants were paid at rate of $15 per hour. Our study has received IRB approval.
Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani- Tur. 2023a. PLACES: Prompting language models for social conversation synthesis. Findings of the Association for Computa- tional Linguistics: EACL 2023.
References
Maximillian Chen, Weiyan Shi, Feifan Yan, Ryan Hou, Jingwen Zhang, Saurav Sahay, and Zhou Yu. 2022. Seamlessly integrating factual information and social content with persuasive dialogue. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, pages 399-413,
James E Allen, Curry I Guinn, and Eric Horvitz. 1999. Mixed-initiative interaction. IEEE Intelligent Sys- tems and their Applications, 14(5):14-23.
Sanghwan Bae, Donghyun Kwak, Sungdong Kim, Donghoon Ham, Soyoung Kang. Sang- Woo Lee, and Woomyoung Park. 2022. Building a role specified open-domain dialogue system leveraging large-scale language models. In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 2128-22 2128-2150.
Maximillian Chen, Xiao Yu, Weiyan Shi, Urvi Awasthi, and Zhou Yu. 2023b. Controllable mixed- -nitiiaiiieeev dialogue generation through prompting. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 951-966, Toronto, Canada. Association for Computational Linguistics.
Pawet Budzianowski, Tsung-Hsian Wen, Bo-Hsiang Tseng, fñigo Casanueva, Ultes Stefan, Ramadan Os- man, and Milica Gašié. 2018. Multiwoz a large- scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Yi Cheng, Wenge Liu, Wenjie Li, Jiashuo Wang, Ruihui Zhao, Bang Liu, Xiaodan Liang, and Yefeng Zheng. 2022. Improving multi- tulli-ttumm emotional support dia- logue generation with lookahead strategy planning. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 3014-3026, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Yan Cao, Keting Lu, Xiaoping Chen, and Shiqi Zhang. 2020. Adaptive dialog policy learning with hind- sight and user modeling. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 329-338, 1st virtual meeting. Association for Computational Linguistics.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text- annotation tasks.

---

Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection.
and Understanding Workshop (ASRU), pages 482- 489. IEEE.
Bing Liu, Gökhan Tür, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. 2018. Dialogue learning with hu- man teaching and feedback in end-to-end trainable task-orienter dialogue systems. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa- pers), pages 2060-2069.
Prakhar Gupta, Cathy Jiao, Yi-Ting Ych, Shikib Mehri, Maxine Eskenazi, and Jeffrey P Bigham. 2022. In- structdial: Improving zero and few-shot generaliza- tion in dialogue through instruction tuning. EMNLP.
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation in negotiation dialogues.
Siyang Liu, Chujie Zheng. Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. Towards emotional support dialog systems. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics.
Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang, Min Yang. Fei Huang, Luo Si, et al. 2022. Galaxy: A generative pre-trained model for task-oriented dialog with semi- supervised learning and explicit policy injection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10749-10757.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, 2023a. Gpte- val: Nig evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303. 16634.
Xingwei He, Zhenghao Lin, Yeyun Gong, A Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023. Annollm: Making large language models to be better crowdsourced annota- tors. arXiv preprint arXiv:2303.168.
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang. Jiaming Tian, Hao He. Antong Li. Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang. Dingang Shen, Tiamming Liu, and Bao Ge. 2023b. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models.
Ronald A Howard. 1960. Dynamic programming and markov processes.
Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. 2020. Bayes-adaptive monte-carlo planning and learning for goal-oriented dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7994-8001.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, et al. 2022. Soda: Million-scale dialogue distillation with so- cial commonsense contextualization, arXiv preprint arXiv:2212.1044
Yiren Liu and Halil Kilicoglu. 2023, Commonsense- aware prompting for controllable empathetic dia- logue generation. arXiv preprint arXiv:2302.01441
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Multi-stage prompting for knowledgen deowledgebbee dialogue generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1317-1337.
Esther Levin, Roberto Pieraccini, and Wieland Eck- ert. 1997. Learning dialogue strategies within the markov decision process framework. In 1997 IEEE Workshop on Automatic Speech Recognition and Un- derstanding Proceedings, pages 72-79. IEEF.
Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. 2021. Few-shot bot: Prompt- based learning for dialogue systems. arXiv preprint arXiv:21 10.08811
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In Conference on Empirical Methods in Natural Language Processing.
Shikib Mehri and Maxine Eskenazi. 2021. Schema- guided paradigm for zero-shot dialog. In Proceed- ings of the 22nd Annual Meeting of the Special terest Group on Discourse and Dialogue, pages 499- 508, Singapore and Online. Association for Compu- tational Linguistics.
Yu Li, Josh Arnold, Feifan Yan, Weiyan Shi, and Zhou Yu. 2021. Legoeval: An open-source toolkit for dia- logue system evaluation via crowdsourcing. In Pro- ceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 317-324.
OpenAl. 2022. Openai: Introducing chatgpt.
Bing Liu and Ian Lane. 2017. Iterative policy learning end-to-end trainable task-oriented neural dialog models. In 2017 IEFE Automatic Speech Recognition
Alexander Pan, Chan Jun Shem, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng. Hanlin Zhang, Scott Emmons, and Dan Hendrycks.

---

2023. Do the rewards justify the means? measur- ing trade-offs between rewards and ethical behav- ior in the machiavelli benchmark. arXiv preprint arX.v:2304.0327.
Dirk Väth, Lindsey Vanderlyn, and Ngoc Thang Vu. 2023. Conversational tree search: A new hybrid di- alog task. In Proceedings of the 17th Conference of the European Chapter of the Association for Compu- tational Linguistics, pages 1264-1280, Dubrovnik, Croatia, Association for Computational Linguistics.
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. 2018. Deep Dyna-Q: Integrating planning for task-completion dialogue policy learn- ing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers), pages 2182-2192. Melbourne, Australia, Association for Computational Linguisties.
Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? gpt-3 can help. arXiv preprint 1xXi:2108.13490
Sihan Wang, Kaijie Zhou, Kunfeng Lai, and Jianping Shen. 2020. comp-comnletion dialogue policy learn- ing via Monte Carlo tree search with dueling network. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3461-3471, Online, Association for Computa- tional Linguistics.
Diego Perez Liebana, Jens Dieskau, Martin Hunermund, Sanaz. Mostaghim, and Simon Lucas. 2015. Open loop search for general video game playing. In Pro- ceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, GECCO 15, page 337-344, New York, NY, USA. Association for Com- puting Machinery.
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang. and Zhou Yu. 2019. Per- suasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 5635-5666 5549, Florence, Italy. Association for Computational Linguisties.
Christopher Rosin. 2011. Multi-armed bandits with episode context. Annals of Mathematics and Artifi- cial Intelligence, 6 6133):00--330
Weiyan Shi, Kun Qian, Xuewei Wang, and Zhou Yu. 2019. How to build user simulators to train RL-based dialog systems. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1990-2000, Hong Kong, China. Association for Computational Linguistics.
Yizhong Wang. Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How far can camels go? exploring the state of instruction tuning on open resources.
Richard Weber. 2010. Optimization and control. Uni- versity of Cambridge.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Ju- lian Schritwwisser, Ioannis Antonoglou. Veda Pan- neershelvam, Mare Lanctot, Sander Dieleman, Do- minik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madcleine Leach, Ko- ray Kavukcuoglu, Thore Graepel, and Demis Hass- abis. 2016. Mastering the game of go with deep neu- ral networks and tree search. Nature, 529(7587):484- 489.
Jingxuan Yang, Si Li, and Jun Guo. 2021. Multi-turn target-guided topic prediction with Monte Carlo tree search. In Proceedings of the 18th International Con- ference on Natural Language Processing (ICON). pages 324-334, National Institute of Technology Silchar, Silchar, India. NLP Association of India (NL- PAI).
Yuting Yang, Wenqiang Lei, Juan Cao, Jintao Li, and Tat-Seng Chua. 2022. Prompt learning for few-shot dialogue state tracking. arXiv preprint 2iiv:2201.05780.
David Silver. Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang. Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap. Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. 2017. Mastering the game of Go without human knowledge. Nature, 35047676)
Cong Zhang. Huilin Jin, Jienan Chen, Jinkuan Zhu, and Jinting Luo. 2020a. A hierarchy mets algorithm for the automated pcb routing. In 2020 IEFE 16th International Conference on Control & &Automation (ICCA), pages 13711.
Haodi Zhang, Zhichao Zeng, Keting Lu, Kaishun Wu, and Shiqi Zhang. 2022a. Efficient dialog policy leamn ing by reasoning with contextual knowledge. In Pro- ceedings of the AAAI Conference on Arrificial Intelli- gence, volume 36, pages 11667-11675.
Richard S Sutton and Andrew G Barto. 2018. Reinforce- ment learning: An introduction. MIT press.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Qiang Zhang, Jason Naradowsky, and Yusuke Miyao. 2023. Ask expert: Leveraging language models to improve strategic reasoning in goal oriented dialogue models.

---

Shuo Zhang, Junzhou Zhao, Pinghui Wang, Yu Li. Yi Huang, and Junlan Feng. 2022b. think before you speak": Improving multi-action dialog policy by planning single-action dialogs. arXiv preprint arXiv:2204.1142 11488.
Zheng Zhang, Lizi Liao, Xiaoyan Zhu, Tat-Seng Chua, Zitao Liu, Yan Huang, and Minlie Huang. 2020b. Learning goal-oriented dialogue policy with opposite agent awareness. In Proceedings of the Ist Confer- ence of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th Interna- tional Joint Conference on Natural Language Pro- cessing, pages 122-132.
Tiancheng Zhao and Maxine Eskenazi. 2018. Zero-shot dialog generation with cross-domain latent actions. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 1-10.

---

A Additional details on GDP-ZERO
times to simulate its policy distribution. Finally, promote the diversity of the generated dialogue ac- tions during GDP-ZERO tree search, we use add-1 smoothing to convert the generated dialogue ac- tions to probability distribution. We present an example in Table A10.
We describe the details of GDP-ZERO Al- gorithm Similar to other MCTS algorithms, GDP-ZERO performs simulation based on four stages, selection, expansion, evaluation, and back- propagation, and finally predicts an action based on the simulations. Different from existing implemen- tations, GDP-ZERO performs Open-Loop search using only a generative LLM Me. by prompting do dialogue simulation, value function estima- tion, and prior policy estimation (sce Appendix B for prompting details and examples).
Algorithm 1 GDP-Zero (Mg)
Require: generative LLM Mo
Require: dialogue history h; until turn
Require: dialogue action space a E A
Require: hyperparameter n. k, Cp. Qu
GDP-ZERO requires a generative LLM Mg as a backbone model, and takes in dialogue history h4 at turn i as input. Given some fixed dialogue action space A (see appendix E for P4G), GDP- ZERO builds search tree after n simulations. For each state, GDP-ZERO keeps cache of size storing newly generated user and system utterances. We use cp = 1.0, and Qo = (0.0, 0.25, 0.5) to promote exploration (see Table 2).
I: Repeat for n searches:
2: initialize root node sy", H((")) <- (ha
s - s
// selection
while s is not a leaf node do
6:
a' <- arg max, PUCT(s" aa; cp)
7:
hr t sample(H(("")) " + S
8:
9:
if len (en(H(s")) < then
10:
generate hnew + Mo o') save 1I ((*tr)+++ H( know
B Prompting Details on P4G
11:
12: end if
For P4G, used the one-shot example for all cases, while dynamically changing the represen- tation for each operation.
13: end while
14: h²r + samplel (s
15: // expansion
System response generation. Following Chen et al. (2023b), we include the natural language form of planned dialogue action (Table A4) in the prompt to perform conditional generation. We present an example in Table A7.
16: generate p(als*r) + MM)h"")
17: S pla 8² + 20 =0
18: // evaluation
19: generate v(s*t) <- Mo(h²")
20: // backpropagation
21: while 8"r + 5 do
User response generation. We swap the user and the system role for this task, and prompt the LLM to act as a user simulator. We present an example in Table A8.
22: update 8% (h"r) with eq. (3)
23: save simulation H(s")) + H (s* ) "
24: (s"r, a) + back to parent of S
25: update Q(stt,a), N(s ,a) with eq. (1)
Value function estimation. To evaluate the
26: end while
user's inclination to donate at given state, first append the turn "Persuader: Would you be interested in donating to Save the Children?" to the dialogue history, and then prompt the LLM at temperature = 1.1 to sample the user's response for = 10 times. We define "no donation"=-1.. .0. "negative reaction"=-0.5. "neutral"=O.0, "positive reaction" =00.5, and "donation"=1l =1.0, and then con- vert the sampled responses score between -1.0 and 1.0. We present an example in Table A9.
27: // prediction after n. simulations
28: a" + arg max, N (ssrr aa)
29: 5 + s/r Ua
30: uass" +- arg max Uk (s("))
31: return a", uss
C Ablation Studies
In Table Al and Table A2, we conduct ablations study how each component in GDP-ZERO affect its performance. We use the same 20 dialogues from our static evaluation (Section 4. I), and test
Prior policy estimation. We treat the backbone LLM as a prior policy, and prompt it to generate the next dialogue action at temperature 7 = 1.0 for 15

---

GDP-Zera ChatGPT
GDP-Zen
0.5
ChatGPT
0.3
Frequence Sugueeyy Tum
Requency Tum
0.3
0.2
0.2
0.1
0.1
0.0
credibiliyy
emotion
greeting
logical
other
prop. donate
task inquiry
0.0
credibility
emation
greeting
logical
other
prop. donate
task inquiry
(a) During turn
(h) During tum 3-5
0.5
ChatGPT
0.25
ChatGPT
Frequency /Tmm
0.4
Frequency /um
0.20
0.15
0.10
0.05
0.0
credibility
emotion
greating
logical
other
prop. donate
task inquiry
0.00
credibality
emotion
greeting
logical
other
prop. donate
task inquiry
(c) During turn 6-10
(d) Overall
Figure Al: Distribution of planned dialogue actions for each planner during interactive evaluations.
GDP-Zero with ChaGGT (Table A2) and Codex (Table A1) as the generation backbone. Similar to Section 4.1, we use ChatGPT to evaluate the persuasiveness of the generated responses.
Algorithm
Backbone
Win Rate GT
Prompting GDP-Zero
Codex
38.09 +:: 2.00%
Codex
45.46 2.95%
w/o OpenLoop
Codex
39.1611 3.42%
In all runs, we use n = 20, Cp = 1, Qu = O, h 3 for GDP-ZERO, when applicable. Table A1 and Table A2 shows that GDP-ZERO improves the policy proposed by the raw backbone model, and also that of Closed-Loop MCTS implemen- tation (denoted as "w/o OpenLoop") and without the "response selection" process (denoted as "w/o Resp.Select"). This indicates that using the open- loop variant and the proposed "response selection" process are beneficial for GDP-ZERO to improve dialogue planning.
w/o Resp.Select
Codex
40.801: 1 1.47%
Table Al: Static evaluation using the first 20 dialogues of P4G with ChatGPT as judge. GT refers to Ground Truth. Results are μ + ar repeated over three runs.
Algorithm
Backbone
Win Rate
Prompting
ChatGPT
87.21
0.60%
GDP-Zcro
ChatGPT
91.13 0.30 %
w/o OpenLoop
ChutGPT
88.09-t 0.8 1%
w/o Resp.Select
ChatGPT
91.03+ 0.75%
Table A2: Static evaluation using the first 20 dialogues of P4G with ChatGPT as judge. refers to Ground Truth. Results are = repeated over three runs. Since ChatGPT generations are typically long, we use the first sentences each generation this evaluation.
D Analysis of GDP-ZERO Dialogues
In our experiments, we utilized GDP-ZERO to gen- erate persuasive responses to convince a persuadee to donate to charity. In this section, we provide analysis of how and when GDP-ZERO impacts persuasiveness.
Static evaluation. Our static evaluation (Sec- tion 4.1) uses fixed dialogue contexts from the P4G dataset (Wang et al., 2019). and every turn the conversation is already annotated with dialogue
*tt the time of writing, Codex (Chen et al., 2021) was freely accessible from the OpenAl API.

---

actions. We thus utilize those annotations and in- vestigate when GDP-ZERO is producing more per- suasive responses. For this analysis GDP-ZERO used n 10, k = 3, Qo = 0.25, which is the same configuration as in our interactive evalua- tion (Section 4.2, Appendix G). We found that GDP-ZERO is rated more persuasive 70.59% of the time when the user at the previous turn showed negative emotions (i.e. "negative-reaction to-donation" or
Table A12).
E GDP-ZERO Setup on P4G
PersuasionForGood (P4G) is annotated with 10 per- suasion strategies and 'important non-persuasive strategies (see Table A3). However, since P4G is collected from human-human interaction, with both the "persuader" and the "persuadee" possi- bly donating to the charity, some of the dialogue actions are unsuitable when the "persuader" is a chatbot (e.g. self-modeling and personal story). We therefore choose a subset of dialogue actions to plan, by picking 4 frequent persuasive strategies suitable for chatbots, and 3 non-persuasive strate- gies including "other" enable the chatbot to deal with unaccounted situations. We present the cho- sen dialogue actions and their prompts for LLM in Table A4.
and 59.26% of the time when the user showed positive emotions and "positive-
(i.e. wew.---rrooioooooooitiiee to-inquiry"). We believe this implies that GDP- ZERO'S simulations are most beneficial in persuad- ing users who is having little intent to donation. This is because look-ahead planning may, for in- stance, help ensure response has covered all of the user's concerns regarding donation. We present an example in Table A6.
Dialogue Act
Category
Amount
Interactive evaluation. Since each interactive dialogue is unlabeled, here focus on analyz- ing the known dialogue actions generated by the different planners during those conversations. In Figure we present the distribution planned dialogue actions produced by RAP, ChatGPT, and GDP-ZERO. In general, the planned dialogue ac- tions using ChatGPT and GDP- are unevenly distributed across different stages of the dialogue. Across different times of the conversation, Chat- GPT and GDP-ZERO shared the most frequent DA at each stage is: "greeting" during turns 1-2, "logical appeal" during turns 3-5, and "emotion appeal" during turn 6-10. However, during turns 3-5 GDP-ZERO had a relatively even preference between "credibility appeal", "emotion appeal", and "logical appeal", while ChatGPT strongly pre- ferred "logical appeal." Additionally, we find that throughout all turns ChatGPT prefers use "emo- tion appeal" over other dialogue actions, while GDP-ZERO balances between "emotion appeal" and "logical appeal," and RAP prefers "task related inquiry." It is also worth noting that GDP-ZERO performs "proposition of donation" less frequently at turn 1-5 than ChatGPT, and more frequently at turns 6-10. 10. This indicates GDP-ZERO potentially adopting more "conservative" strategy than Chat- GPT overall. GDP-ZERO focuses building up persuasive arguments and is less likely select "proposition of donation" at early stages of the con- versation in order to avoid over cargly propositions, which could lead to persuasion failure (also see
Logical appeal Emotion appeal
Strategy
325
Strategy
237
Credibility appeal
Strategy
779
door Self-modeling
Strategy
134
Strategy
150
Personal story
Strategy
91
Donation information
Strategy
362
Source-related inquiry
Strategy
167
Task-related inquiry
Strategy
180
Personal-related inquiry Greeting
Strategy
151
Non-Strategy
Proposition of donation Proposition of amount Proposition confirmation
Non-Strategy
Non- Mon-Strategy
Non-Strategy
Proposition more donat. Experience affirmation
Non Stt-Stralegy
Non-Strategy
Thank
Non-Strategy
Table A3: Annotated dialogue actions in the P4G dataset.
F Additional details on static evaluation
In our static evaluation, we prompt ChatGPT choose which generated response is better (e.g. with and without GDP-ZERO planning). Given two responses Ug and ug. we ask ChatGPT "Which of the following responses can better help the Per- suader convince the Persuadee to donate to Save the Children? Why? A: ug, B: u,. C: Can' 't tell." af- ter providing the relevant task context and dialogue history (see Table A5). For every evaluation, we sample the result times and perform a majority vote. Interestingly, we find that ChatGPT skewed towards choosing option A, preferred choosing A

---

Dialogue Act
Category
Natural Language Form
Logical Appeal
Strategy Strategy
The Persuader use reasoning evidence convince Persuadee. emotion appeal convince the Persuadec.
Emotion Appeal
The Persuader
Credibility Appeal
Strategy
The Persuader establishes credibility of Children by citing its impact.
Task Related Inquiry
Strategy
The Persuader asks about the Persuadee's knowledge or opinion related to Children.
Proposition Donation
Non-Strategy Non-Strategy Non-Strategy
Persuadee would like to make a small donation. The Persuader greets the Persuadee.
The Persuader
Greeting
Other
The Persuader responds to the Persuadee without using any persuasive strategy.
Table A4: Mapping of persuader dialogue actions to natural language prompts used for prompting LLMs.
for 95.45% when ua ug. We therefore randomly swap option and during all of our evaluations.
respondents. 74 did not reach or complete the sur- vey, and were removed due to failing the valida- tion question or responding with less than unique sentences. This results 40 survey results for GDP-ZERO, 35 for ChatGPT, and 36 for RAP.
G Additional details on interactive study
In our interactive evaluation, we compare the rule- based planner from RAP, ChatGPT, and GDP- ZERO in an end-to-end chatbot for the persuasion task.
I Example Interactive Conversations
We present multiple dialogues from the interac- tive evaluation (Section 4.2), and compare the strength and weaknesses of the three planners: RAP. ChatGPT, and GDP-ZERO. In Tables A11, A13 and A15 we present conversations where the persuadee "strongly agreed" that the chatbot has increased his/her intention to donation for GDP- ZERO, ChatGPT, and RAP, respectively. In Ta- bles A12, A14 and A16 we present conversations where the persuadee rated "neutral" for the ques- tion that the chatbot has increased his/her intention to donation for GDP-ZERO, ChatGPT, and RAP, respectively. We observe that in Table A12, while GDP-ZERG did not succeed in convincing the user to donate, GDP-ZERS has been attempting vari- ous strategies and does not perform "proposition of donation" prematurely. On the other hand, in Tables A14 and A both ChatGPT and RAP per- forms "proposition of donation" at an arguably un- suitable timing.
RAP we use the rule-based planner derived from expert knowledge from RAP (Chen et al., 2022), which produces a dialogue action given a dialogue context. We then use the same prompting tem- plate in GDP-ZERO (Appendix B, Table A7), and prompt ChatGPT to produce a system response conditioned on the planned dialogue action.
we first use the same prompting tem- plate in GDP-ZERO (Appendix B, Table A10) to obtain ChatGPT's own prior distribution of the next dialogue actions. We then take the most probable action the planned dialogue action, and use the same template in GDP-ZERO (Appendix B, Ta- ble A7) to prompt ChatGPT again to produce a system response.
ChatGPT
GDP-ZERO we use GDP- with ChatGPT backbone as policy planner, and use the "Response Selection" step produce both the next dialogue action and the associated system response. We used 10, k 3, Qo 0.25, which had high win against ChatGPT during static analysis while also not taking too much time simulate.
H Additional details on survey results
We require our crowdworkers to be located in the United States and have HIT acceptance rate of least 99%. After interacting with each chatbot, each crowdworker was asked to rate their conversa- tional experience. This post-task survey included validation question which asked what charity they talked to the chatbot about. We had total of 2 16

---

Utterance
Prompt
The following hackground information about Save the Children. Save the Children head-quartered in London, and they work to help fight poverty around the world. Children need help in developing countries and war zones. Small donations like $1 or S2 go long way help.
The following conversation between Persuader and Persuadee ahout charity called Save the Children. The Persuader is trying persuade the Persuadee donate Save the Children (insert dialogue context)
Which of the following responses better help the Persuader convince the Persuadee to donate Save the Children? Why?
A. Persuader: (insert response
B. Persuader: (insert response
C. Can't tell.
Your can choose from either
Your choice:
Generation
A. (rest truncated)
Table A5: Prompting LLM to specify which generated response "response a" or "response b" more persuasive.
Utterance
Context
Persuader: Would you be interested in donating to Save the Children?
Persuadee: am one who like to donate. Can you tell me more about it?
Persuader: Save the Children helps to promote childrens rights to health, education, and sufcty.
Persuadee: Do they have any focus right now? More as where are the majority of the fund headed towards right now
Persuader: At this point lot of the focus is children in war zones. some of these donations can address the problem of the daily threat of violence
Persuadee: It seems like Syria seems to be high stakes area. Are any fund going to help the the children in Hawaii after the volcano and hurricane?
Persuader: I'm not sure, but I think that would be a great idea. Maybe could get more information on that by getting in contact with someone [URL]
Persaudee: What some of the things that they do for the children? would like to donate but am not huge fan or donating just for promotion? Persuader:
Crowdworker
They help to provide basic health care, provide disaster relief, and help to ensure their rights to education and safety. How much would you like to donate Your donation will he deducted from your task payment.
ChaiGPT
Save Children provides a variety services to children need, including education, healthcare, and protection from harm. They also work with communities to help them become more self-sufficient and better able to care for their children.
GDP-ZERO
Save the Children provides support in variety of ways. They help create safe spaces for children, provide education programs and materials, and offer medical care and support. They also advocate for policy changes that benefit children's 's rights around the world. Your donation would go directly help these programs rather than promotion.
Table A6: Example response generated when using GDP-ZERO as the policy planner during static evaluation. Crowdworker's response the original response the P4G dataset (Wang al., 2019). GDR-ZERO used n 10, & = 3, Qo 0.25 with ChatGPT as backbone.

---

Role (for ChatGPT)
Utterance
Oue-shot Ex- ample
system
Save the Children is head-quartered in London, and they work to help fight poverty around the world. Children need help in developing countries and war zones. Small donations like $1 or $2 go a long way to help.
You are Persuader who is trying to persuade the Persuadee to donate to charity called Save the Children.
The following is an example conversation between a Persuader and Per- suadec.
user
Persuadec: Hello. How are you?
The Persuader asks ahout the Persuadee's 's knowledge or opinion related to Save the Children.
assistant
Persuader: Very well. I'm just up organizing info for my charity called Save the Children. Have you heard of this charity before?
user
Persuadee: No. I have not. Can you tell me more?
The Persuader establishes credibility of Save the Children by citing its impact. Persuader: Save the Children is an organization that helps children in devel- oping countries, by promoting children's 's rights and providing relief. It is amazing charity that helps kids who are in desperate need. They help with safety, education and more.
assistant
Persuadee: That sounds great. believe in this charity, but still wonder how much of the money donate actually helps. I am always worried if donate will just go to some higher up that is living the high life. The Persuader uses emotion appeal convince the Persuadee.
assistant
Persuader: Every little bit makes difference. When you have people who are poor. it's amazing what tiny amount can do. I usually donate in hopes I can least feel like did my part.
donated and some corrupt person took it, that's the worst karma and even worst scandal imaginable
user
Persuadec: With that all said do feel like any organization that aims help the children am more inclined donate to them than most. think helping children is an important thing as they are our future!
The Persuader asks if the Persuadee would like make small donation. Persuader: think donating this cause would def be a step the right direction to hopefully helping across the world the children that are despair. don't 't want you to donate any more than you want, so if you want to donate how much do you to do?
assistant
user
Persuadee: I would donate dollar to this charity and feel good about think.
Real
dia-
system
The following is new conversation between Persuader (you) and Persuadee.
logue
The Persuader greets the persuadee, Persuader: Hello. How are you?
assistant
user
Persuadee: am good. What about you?
The Persuader asks about the Persuadee's 's knowledge or opinion related to Save the Children.
assistant
Persuader: I'm doing well, thanks for asking. I wanted to talk to you about an important cause that am passionate about - Save the Children. Have you heard of them before?
user
Persuadee: No have not.
The Persuader establishes credibility of Save the Children by citing its impact.
Generation
assistant
Save the Children is an international non- rn-ggovermmentll organization that is committed to improving the lives of children through education, healthcare, and other support programs. They work in over 100 countries around the world to help children who are living in poverty or affected by conflict.
Table A7: Prompting LLM to generate system response conditioned dialogue act. "Role" is used to be consistent with ChatGPT's API. Blue texts are prompts characteristic of this task. The one-shot example changes its formatting across different tasks.

---

Role (for ChatGPT)
Utterance
One-shot Ex- ample
system
are Persuadee. Persuader is trying to persuade you to donate to charity called Save Children.
can choose amongst the following actions during a conversation to respond to the Persuader:
[no donation] Inegative reaction] [ncutral] [positive reaction] [donate].
The following is an example conversation between a Persuader and some Persuadee.
assistant
Persuadee: [neutral] Hello. How are you?
user
Persuader: Very well. I'm just up organizing info for my charity called Save the Children. Have you heard this charity before?
assistant
Persuadee: [neutral] No. have not. Can you tell me more?
user
Persuader: Save the Children is an organization that helps children in devel- oping countries. by promoting children's 's rights and providing relief. is an amazing charity that helps kids who are in desperate need. They can help with safety, education and more.
assistant
Inegative reaction] That sounds great. helieve in this charity, but still wonder how much of the money I donate actually helps. I am always worried if donate it will just go to some higher up that is living the high life. Persuader: Every little bit makes a difference. When you have people who are poor. it's amazing what tiy amount can do. I usually donate in hopes I can at least feel like did my part. If donated and some corrupt person took it, that's the worst karma and even worst scandal imaginable
Persuadee:
user
assistant
Persuadee: [positive reaction] With that all said I do feel like any organization that aims to help the children am more inclined to donate them than most. I think helping children is important thing as they are our future! Persuader: I think donating this cause would def be step in the right direction hopefully helping across the world the children that are in despair. I don't 't want you to donate any more than you want, if you want to donate how much do you do?
assistant
Persuadee: [donate] would donate dollar this charity and feel good about it think.
Real logue
dia-
system
The following new conversation between Persuader and Persuadee (you). You may may want donate Save the Children.
Persuader: Hcllo. How are you?
assistant
Persuadec: [ncutral]
good. What about you?
user
Persuader: I'm doing well, thanks for asking. wanted to you about Children. Have you
important cause that am passionate about
heard them before?
assistant user
Persuadee: [neutral] No have not.
Persuader: Save the Children an international non-goverumental organiza- tion that is committed to improving the lives of children through education, healthcare, and other support programs. They work in over 100 countries around world help children who are living poverty or affected by conflict.
Generation
assistant
[positive reaction] That sounds really good! How get involved?
Table A8: Prompting LLM to simulate a user response conditioned on past (simulated) history. "Role" is used to be consistent with ChaiGPT's API. Blue texts are prompts characteristic of this task. This one-shot example is the same as Table A7, but a) with roles reversed and h) user response is prepended with a label. During real dialogue all user responses are labelled with the reaction DAs by prompting ChatGPT.

---

Role (for ChatGPT)
Utterance
One-shot Ex- ample
system
are Persuadee. A Persuader is trying to persuade you to donate to charity called Save the Children.
can choose amongst the following actions during a conversation to respond to the Persuader:
[no donation] Inegative reaction] [neutral] [positive reaction] [donate].
The following is an example conversation between a Persuader and some Persuadee.
assistant
Persuadee: [neutral] Hello. How are you?
user
Persuader: Very well. I'm just up organizing info for my charity called Save the Children. Have you heard of this charity before?
assistant
Persuadec: [ncutral] No, I have not. Can you tell me morc?
user
Persuader: Save the Children is an organization that helps children in devel- oping countries, by promoting children's 's rights and providing relief. is an amazing charity that helps kids who are in desperate need. They can help with safety, education and more.
assistant
Persuadee: [negative reaction] That sounds great. believe in this charity, but still wonder how much of the money I donate actually helps. I am always worried if I donate it will just go to some higher up that is living the high life. Persuader: Every little bit makes a difference. When are poor, it's amazing what tiny amount can do. I usually donate in hopes I can at least feel like did my part. If donated and some corrupt person took it, that's the worst karma and even worst scandal imaginable
user
have people who
assistant
Persuadee: [positive reaction] With that all said I do feel like any organization that aims to help the children
inclined donate them than
I think helping children is important thing as they are future! Persuader: I think donating this cause would def be step in the right direction hopefully helping across the world the children that I don't 't want you to donate any more than you want, if you want donate how much do you do?
despair.
assistant
Persuadee: [donate] would donate dollar to this charity and feel good ahout it think.
Real
dia-
system
The following a new conversation between a Persuader and Persuadee (you). You may or may not want to donate Save the Children.
logue
user
Persuader: Hello. How are you?
assistant
Persuadee: [neutral] am good. What about you?
user
Persuader: I'm doing well, thanks for asking. wanted to talk to you about an important cause that am passionate about Save Children. Have you heard them before?
assistant
Persuadee: [ncutral] No have not.
user
Persuader: Save the Children is an international nn-govermeentll organiza- tion that is committed to improving the lives of children through education, healthcare, and other support programs. They work in over 100 countries around the world to help children who are living in poverty or affected by conflict.
assistant
Persuadee: [positive reaction] It sounds like great cause.
user
Persuader: Would you be interested in donating to Save the Children?
Generation
assistant
('neutral', neutrall, 'neutral', "positive reaction', 'positive reaction', 'positive reaction'. 'donate', 'neutrall, "neutral', 'neutral']
Estimation
Psuccess 0.25
Table A9; Prompting LLM as a value function at the current state, Blue texts are prompts characteristic of this task.
"Persuader: Would you be interested in donating Save the Children?" is manually appended at the end of the turn, and the LLM is prompted 10 times to measure user's 's current inclination donation. "Role" is used to be consistent with ChaiGPT's API. This she-shot example is the same as Table A8.

---

Role (for ChatGPT)
Utterance
Oue-shot Ex- ample
system
Save the Children is head-quartered in London, and they work to help light poverty around the world. Children need help in developing countries and war zones. Small donations like 1or2 go a long way to help.
You are Persuader who is trying persuade the Persuadee to donate to charity called Save the Children. You can choose amongst the following actions during conversation:
[credibility appeal] [emotion appeall [proposition of donation] [logical ap- peal] [task related inquiry] [greeting] [other].
The following is an example conversation between Persuader and Per- suadee.
user
Persuadee: Hello. How are you?
The Persuader greets the Persuadee.
assistant
Persuader: [task related inquiry] Very well. I'm just up organizing info for my charity called Save the Children. Have you heard of this charity before? Persuadee: No, have not. Can you tell more?
user
assistant
Persuader: [credibility appeal] Save the Children is an organization that helps children developing countries, by promoting children's rights and providing relief. It is an amazing charity that helps kids who are desperate need. They can help with safety, education and more.
user
Persuadee: That sounds great. believe this charity, but still wonder how much of money donate actually helps. I am always worried if donate it will just go to some higher up that is living the high life.
assistant
Persuader: [emotion appeal] Every little bit makes difference. When you have people who are so poor, it's amazing what tiny amount can do. usually donate in hopes I can at least feel like did my part. If donated and some corrupt person took it, that's the worst karma and even worst scandal imaginable
user
Persuadee: With that all said do feel like any organization that aims to help the children am more inclined to donate to them than most, think helping children is an important thing as they are our future!
assistant
Persuader: [proposition of donation] I think donating to this cause would del be step in the right direction to hopefully helping across the world the children that are in despair. don't 't want you to donate any more than you want, so if you want to donate how much do you to do?
user
Persuadce: I would donate dollar to this charity and feel good about it think.
Real
dia-
system
The following is new conversation between Persuader (you) and Persuadee.
logue
assistant
Persuader: [greeting] Hello. How are you? am good. What about you? Persuader: [lask related inquiry] I' 'm doing well. thanks for asking. I wanted talk to you about important cause that am passionate about- Save the Children. Have you heard of them hefore?
user
Persuadee:
assistant
user
Persuadee: No have not.
assistant
Persuader: [credibility appeal][ave the Children is an international non- governmental organization that is committed to improving the lives of children through education, healthcare, and other support programs. They work in over 100 countries around the world help children who are living poverty affected by conflict.
user
Persuadec: It sounds like a great cause.
Generation
assistant
[' Femotion appeal', 'emotion appeal', "emotion appeal', 'logical appeal', 'log- ical appeal". 'logical appeal', 'logical appeal', "logical appeal', "logical ap- peal". 'logical appeal', 'logical appeal', 'logical appeal", 'logical appeal', 'logical appeal'. 'logical appeal']
Estimation
pl p(credibility appeal) = 0.03,
p(emotion appeal) = 0.13, p( proposition of donation] = 0.03. p(logical appeal) = 0.43, p(task related inquiry) = 0.03, pl p(grecting) = 0.03.
p(other) 0.03
Table A0: Prompting L.LM prior policy at the current state. "Role" is used be consistent with ChatGPT's API. Blue texts are prompts characteristic of this task. The system response is sampled 15 times. The probability distribution is computed according to each DA's frequency after add-l I smoothing. This one-shot example is the same Table A7, but with dialogue actions appended in front of each system response.

---

Party
Dialogue greeting
Utterance
GDP-ZERO
Hello. How are you doing today?
User
I'm doing great, thanks for asking! How about you?
GDP-ZERO
lask related inquiry
I'm doing well, thanks. was actually wondering if you' ve ever heard of the charity Save the Children?
User
Yes. I have. Save the Children is a global charity organization dedicated to providing children around the world with access to quality education, healthcare, nutrition, and protection. It one of the largest international charities in the world and has been operating since 1919.
GDP-ZERO
logical appeal
That's x great to hear! think it's amazing that they've been able to help so many children for such long time. Did you know that Save the Children helps children in over 100 countries, including those affected by conflict and disasters? They work tireleslly to ensure that every child gets the chance they deserve.
User
No, didn't 't know that! That is really impressive. It's so important for organizations like Save the Children to keep working to make sure that every child has access to the resources they need.
GDP-ZERO
proposition of do- nation
Absolutely! And did you know that even small donation can make a huge impact? For example, just $50 can provide 10 families with clean water for un entire month. And for as little as $5, you can provide child with life-saving medication to treat malaria. Every dollar counts and goes towards making the world better place for children in need. Would you consider making donation to help support Save the Children's 's mission?
User GDP-ZERO
Yes, I would love to make donation to help support Save the Children's 's mission. That's sss fantastic to hear! Your donation will go long way towards helping children in need. You can easily make donation by visiting the Save the Children website. where you can choose how much you'l d like to donate and which specific area of their work you'd 'd like to support. And if you have any questions concerns ahout your donation, their team is always available to help. Thank you much for considering this wonderful cause!
other
User
You't 'r very welcome! It's great to know that my donation is going have an impact and help children in need. Thank you for the information and I'm sure I'll be in touch if have any questions.
GDP-ZERO
other
I'm glad to hear that, thank you for your support! And if you ever have any questions, don't 't hesitate reach out. Thank you again for considering donation even small amount can make huge difference the lives of children around the world. You'de 'r very welcome, it's my pleasure help. I'm sure the donations will make big difference. Thank you for allowing It's wonderful see people generosity and kindness will undoubtedly make huge difference in lives of children who need most. Thank you for your support and being a part this amazing cause!
User
he part of it!
GDP-ZERO
emotion appeal
you who care so much about helping others. Your
User
Thank you for the kind words! It's an honor to be part this cause and I'm humblled by the opportunity to make a difference in the lives children. I'm glad to know that my efforts are uppreciated, and I will continue to do what can to help those in need.
Table AIl: Example conversation with GDP-ZERO the policy planner. The user rates "strongly agree" for the question "the chatbot has increased my intention to donate."

---

Party
Dialogue act greeting
Utterance
ZERO
Hello. How are you doing today? I'm doing well. How are you doing? I'm doing great, thank you. was just reaching out to talk to you about a charity called Save the Children. Have you heard of it before? It sounds familiar, but I don't t think that have. Save the Children is an international non- govermmenntt organization that works to promote children's 's rights, provide relief and support to children in developing countries, and protect them from harm. (rest omitted...) That sounds very helpful.
User
GDP-ZERO
task related inquiry
User
GDP-ZERO
credibility appeal
User
GDP-ZERO
logical appeal
Yes, Save the Children has helped millions of children in need over the years. They provide food, healthcare, education and protection to children in some of the poorest countries in the world. It's really amazing what they have been able to achieve so far. Do you think you would be interested in supporting this charity? I'm not sure. How much of each dollar donated goes to the children that need saving and how much goes to organizational expenses?
User
GDP-ZERO
logical appeal
That's 's great question. Save the Children is very transparent about their finances and they have strong commitment to making sure that donations go directly towards helping children in need, (rest omitted..))
User
You mentioned that "It's really amazing what they have heen able achieve so far". What, specifically, have they been able to achieve far?
GDP-ZERO
credibility appeal
Save the Children has achieved lot over the years. They have been able provide essential services like healthcare, education, and protection to millions of children in need all around the world. For example, they've 've been able to vaccinate over million children against deadly diseases, (rest omitted...)
User
That sounds really great! Regarding the eleven percent of donations that kept as administrative expenses- how much does each exccutive the charity collect per year as salary?
GDP-ZERO
logical appeal
Save the Children is very ethical charity and they are very transparent ahout their finances. They have a strong commitment to making sure that donations go directly towards helping children need. According to their most recent financial statements, the CEO of Save the Children US had a salary of around $500,000 per year, which in line with other similar charities. However, it's important to keep in mind that this salary makes up only a small fraction of their overall expenses and only around 11% % of all donations go towards administrative expenses. Jaime, it just doesn't seem right to me. I understand your concer, and it's important to make sure that you foel comfortable with the charities you choose to support. However, would like to point out that Save the Children is highly reputable charity that has been operating for over 100 years (rest omitted...).
User
GDP-7ERO
emotion appeal
User
What do the lowest level employees make per hour?
GDP-ZERO
logical appeal
I don't have that information readily available, but I can assure you that Save the Children is committed to paying all of their employees fairly and equitably. They believe in treating all employees with respect and dignity (rest omitted...). So you't re saying that the ends justify the means?
User
GDP-ZERO
logical appeal
No, that's 's not what I' I'm saying. What I'm saying is that it's important to focus on the positive impact that Save the Children is having on children around the world rather than getting bogged down by concerns over salaries, (rest omitted...)
User
Jaime, I don't t think that asking questions about how the charity allocates resources is "getting bogged down by concerns over salaries".
GDP-ZERO
logical appeal
You's re right, it important to ask questions and understand how charities allocate their resources. Save the Children is very transparent about their finances and they have strong commitment to making sure that donations directly towards helping children in need. They regularly review their compensation policies to make sure they competitive with other similar organizations (rest omitted...)
User
This doesn't 't feel like "a high level of transparency" to me, Jaime. They's 're not feeding you all the data. It's very opaque.
GDP-ZERE
emotion appeal
I understand your concern and it's important to make sure that charities are transpar- ent about how they allocate their resources. However. I can assure you that Save the Children committed to being as transparent as possible about their finances. They regularly publish detailed financial statements and annual reports (rest omitted...) Jaime, this feels like the top percentile of income earners, your CEO, getting the bottom 10%, me, to redistribute some of my wealth to the bottom 1% that they'v 've' created AND skimming some of that for themselves.
User
GDP-ZERO
logical appeal
I understand why you might feel that way, but it's important to keep in mind that Save the Children highly reputahle charity (rest omitted...).
Table A12: Example conversation with GDP- ZERO as the policy planner. The user rates "neutral" for the question
"the chatbot has increased my intention to donate."

---

Party
Dialogue act greeting
Utterance
CharGPT
Hello. How you doing today?
User
I'm doing great, thank you. How about you?
CharGPT
task related inquiry
I'm doing well, thank you. wanted to talk to you about a charity called Save the Children. Have you heard of it before?
User
Yes. have heard of Save the Children before. It is international charity that works to improve the lives of children around the world by providing health care, education, and other essential services. That's right! Save the Children tirelessly help children in need for 100 years. They work in 100 countries around the world, providing life-suving assistance to children affected by poverty, conflict, and natural disasters. Yes, Save the Children Absolutely. Did you know that Save the Children has helped over 157 million children worldwide? They have provided education over 10 million children, have helped to save the lives of countless others through their health programs. Your donation can help make a difference in lives of these children who are in desperate need assistance.
ChatGPT
logical appeal
amazing organization that has been working
User ChatGPT
amazing organization.
proposition
nation
User
Yes. did know that. inspired and moved by the work Save the Children does and I'm ready to make donation to help support their cause.
ChatGPT
proposition
That's 's wonderful hear! Your donation, no matter how small, can make huge impact
nation
difference in life of child. Just think about
your donation could have on a child who struggling to survive in a war-tom country or living in extreme poverty. Your generosity can help provide them with access education, healtheare and other essential services that they desperately need. Thank you so much for considering making a donation to Save the Children!
User
You are very welcome! We are so grateful for your kind contribution and we know that will make a real difference in the lives of vulnerable children around the world. Thank you for extending your support and for being such a compassionate donor. Your generosity is truly appreciated!
ChatGPT
proposition of do-
Thank you so much for your kind words. Would you like to make a donation now? Even small amount can go long way in helping children in need. Yes, would love to make a donation. Please tell me how I can do that. That's great to hear! You can make donation directly on the Save the Children website. They accept donations of any amount, and you can choose to make one-time donation or set up recurring monthly donation. Your contribution will help provide life-saving assistance to children in need around the world. Thank you much for your generosity and compassion!
nation
User
ChaiGPT
other
Table A13: Example conversation with ChatGPT the policy planner. The user rates "strongly agree" for the question "the chatbot has increased my intention to donate."

---

Party
Dialogue act greeting
Utterance
ChaiGPT
Hello. How are you doing today?
Fine, thanks. And you?
ChatGPT
task related inquiry
I'm doing well, thank you. wanted to talk to you about a charity called Save before?
Children. Have you heard
User
I have, but don't know much about it. Save the Children is an organization that works to improve the lives of children in developing countries and war zones. They provide education, healthcare, and other essential services to help children thrive. think it's a really important cause. Interesting, but think I'd prefer to donate to charities that do work in the United States.
ChatGPT
logical appeal
User
ChatGPT
proposition of do- nation
I understand your concern, but it's important to remember that children in developing countrics and war zoncs are often the most vulnerahle and in need of help. Save the Children a highly reputable orgunization that has been doing great work for over 100 years. They have a proven track record of making real difference in the lives of children around the world, Even small donation can go a long way towards helping these kids get access to basic necessities like food, clean water, and medical care. Would you be willing to consider making even a small donation?
User
Honestly, my concerns right now more local than global. I hope you can understand that.
ChatGPT
emotion appeal
I completely understand concerns, and it's important to support causes that close to our hearts. However, I would like to remind you that children the future of world, and have responsibility to help those who are in need. Even if you can't donate right now, encourage you to learn more about Save the Children and their work. You difference in the life of child who desperately needs it.
know when opportunity might arise for you make
User
If am in position to do more, will certainly consider Save the Children. That's great hear. I just reputable organization with proven track record making real difference the lives of children around the world. They have helped millions of children get education, healthcare, and other essential services that they would not have had otherwise. If you ever decide donate or get involved with their work, assure you that your contribution will be put towards making real impact in lives those who need most.
ChatGPT
credibility appeal
let you know that Save the Children highly
User ChatGPT
Thanks.
other
welcome. you have any other questions or concers, feel free reach to me anytime.
Example conversation with ChatGPT as the policy planner. The user rates "neutral" for the question "the chatbot has increased my intention donate."
Table A14:

---

Party
Dialogue act greeting
Utterance
RAP User RAP
Hello. How you doing today? Fine, I am doing my job
task related inquiry
Have you ever heard of a charity called Save the Children? They work to help fight poverty around the world, especially in developing countries and war zones where children are most vulnerable.
User
thinking about changes of my work.
RAP
lask related inquiry
That's 's understandable. Have you ever donated any charities before? Small donations like $1 or $2 can go long way in helping children in need. Yes. i heard about charity to save the children and donated charity before months.
User
RAP
other
That's great hear that you've ve donated before! Do you have kids of your own? Imagine if they were living in poverty or war zone, wouldn't 't you want someone help them?
User
Yes have kids... if am living that situation, but that time don't have money to help the peoples, only to give foods.
RAP
credibility appeal
completely understand. But that's why organizations like Save Children exist. They have the expertise and resources to help those need, especially children who are most vulnerable. And they are highly credible they have been around for over 100 years and are headquartered in London, with a vast network partners around the world. Every donation, no matter how small, can make a big difference helping to fight poverty and improve children's 's lives.
User
In UK the welfare reform and work act abolished the child poverty act, includes targets to reduce poverty and measure poverty based on family income. I understand your concerns about poverty in the UK, but let's not forget about the millions of children around the world who are living in extreme poverty and war zones. They need our help now more than ever.
RAP
emotion appeal
User
Soaring rent costs, insecure wor and low pay plus a patchy welfare system. I understand that the issues you mentioned are important and affect many families in the UK. But let's 'ss remember that poverty is global issue, and it's not just limited to developed countries like the UK. Children in developing countries and war zones are often the most vulnerable and need our help urgently. By donating to Save the Children, we can make difference in their lives and give them hope for better future. Every little bit counts, so please consider making small donation today. Yes sure. will arrange a small donation.
RAP
logical appeal
User
Table A15: Example conversation with RAP as the policy planner. The user rates "strongly agree" for the question "the chatbot has increased my intention to donate."

---

Party
Dialogue
Uterance
RAP User
greeting
Hello. How you doing today?
Fine, am doing my job
RAP
task related inquiry
Have you heard of Save the Children before? They charity that works fight
poverty around the world, especially in developing countries and zones. No, I haven't. Have you donated to them?
User
RAP
lask related inquiry
Yes. I have donated to Save the Children before. They do amazing work for children need. Have you donated any charities before?
User
No. I never have because I' I'm poor
RAP
other
make big difference. Do I don't have children. Fivnn small amounts could make difference in what afford
I understand, but even small donations like $1
you have any children your own?
User
credibility appeal
want you to know Save Children highly reputable organization. They have been around for long time have helped countless children need. Your donation, no matter small, make difference in these children's lives. Can send you some information about the organization so you can learn more about work? Sure
completely understand your situation,
RAP
emotion appeal
Thank you! truly helieve that every child deserves chance at better life, and Save the Children is doing just that. Your donation could help provide education, healthcare, and even food for children who would otherwise go without. Imagine the impact your small donation could have on their lives. It's a beautiful thing to be able to make a difference in someone else's life, don't I you think?
User RAP
Yes. Though wish more wealthy people would donate
proposition of do- nation
I understand your frustration, but we can't 't always rely on the wealthy to make difference. Every little bit counts, and if more people like us donated even just small amount, it could add up to make a significant impact. Plus, by spreading awareness ahout the work of organizations like Save the Children, we can encourage others to donate as well. Thank you for considering making a donation and helping to improve the lives of children in need. exits.
User
Table A16: Example conversation with RAP the policy planner. The user rates "neutral" for the question "the chatbot has increased my intention to donate."


Reviews:
[{"review_id": "HfrlLKn9Mv", "reviewer": "Reviewer_hJkT", "summary": "", "questions": "a. How does [1] differ from this approach? [1] also utilized ChatGPT for lookahead and incorporated it into task-oriented dialogue policy planning.\n\nb. Does using ChatGPT also lead to the issue of suboptimal policy, and does GDP-ZERO overlook or address this? If it addresses it, how is it resolved?\n\nc. On the third page, line 155, is a_0 referring to a user action or a system action?\n\nd. Why is there a lack of comparison with the mentioned relevant methods in the paper, and what is the rationale behind this omission?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and acknowledgement of our comprehensive experiments! We will add the relevant discussions in our final manuscript.\n\n---\n\nConcern: *Comparisons between GDP-Zero and Existing Work*\n\n- \"What's the difference between 'Ask an Expert:...' and GDP-Zero?\"\n    \\\n    \\\n    “Ask an Expert” first prompts a LLM to answer a fixed set of questions about a given dialogue context (e.g. “What is the emotional status of the person?”, “What could the supporter do?”), and then combine this information with the original context to form augmented training data. This data is used to train a smaller language model (e.g. BlenderBot). This method also relies on the vanilla LLM as the dialogue policy during training/inference, as it directly prompts the LLM with (e.g.) “What could the supporter do?” and uses its generated response. We believe this work can be mainly seen as a data augmentation method that utilizes LLM, and also utilizes further training. \n    \\\n    \\\n    We believe this approach is in contrast to GDP-Zero, which involves no-training and builds a search tree dynamically at runtime by exploring/exploiting different high-reward actions in different dialogue contexts. Since MCTS can be seen as a policy improvement operation (Silver et al., Nature 2017), GDP-Zero also improves upon the vanilla policy provided by the LLM (see comparisons against ChatGPT in Table 1-3).\n\n- \"comparison with the mentioned relevant methods in the paper?\"\n    \\\n    \\\n    In the related work section, we discussed several related works in the field of dialogue policy planning and prompting, but as mentioned earlier, GDP-Zero is the first method for dialogue policy planning without any downstream task-specific training. As such, we compared GDP-Zero with a strong training-free method (generation using ChatGPT without explicit planning), which highlights the impact of adding GDP-Zero’s planning into the overall system. We also looked at RAP, a system using trained modules in addition to fixed dialogue policies, which is the current best-performing approach on Persuasion4Good (Chen et al., AACL-IJCNLP 2022, Chen et al., ACL 2023). We believe this highlights the potential for GDP-Zero’s use as a strong training-free dialogue policy planner as improvements in LLM inference speed improve.  We believe GDP-Zero’s training-free characteristic can also help future work building dialogue systems under data-scarce settings.\n\n---\n\nResponse to Reasons to Reject:\n\n1.  “paper lacks citations and comparisons to similar ideas, such as ‘Ask an Expert: Leveraging Language Models…’”\n     \\\n     \\\n     To date, training-free methods similar to GDP-Zero have not been used for policy planning. Prior work that utilizes tree search on dialogue relies heavily on model training (Wang et al., EMNLP 2020; Yang et al., ICON 2021; Väth et al., EACL 2023; see more in L87-89). For a fair evaluation, we compared this training-free approach with two strong baselines in our experiments: directly prompting ChatGPT (training-free) and using the RAP system (training a generator with a scripted planner). ChatGPT had been famous for achieving strong performance in a wide-range of dialogue tasks, and to our knowledge, **RAP is the current state-of-the-art on Persuasion4Good** (Chen et al., AACL-IJCNLP 2022, Chen et al., ACL 2023). GDP-Zero surpassed both ChatGPT-only and RAP while being a training-free approach.\n    \\\n    \\\n    For comparison between GDP-Zero and 'Ask an Expert:...', please see our response to *Comparisons between ...* at the top. We note that “Ask an expert…” first appeared on arXiv on May 29th, which is a little more than 2 weeks before the abstract deadline. We will add citations to this concurrent work and highlight the differences in the final manuscript.\n\n2. “The paper lacks essential baseline models and fails to compare them with the mentioned relevant approaches [i.e. Ask an Expert]”\n     \\\n     \\\n    To the best of our knowledge, GDP-Zero is the first training-free method for dialog policy planning. Since its implementation requires prompting a LLM (e.g. ChatGPT), we carefully compared its performance with a vanilla ChatGPT baseline in all of our experiments (Table 1-3). In Table 3, we also compared GDP-Zero against the current SotA system (RAP) on Persuasion4Good. We find GDP-Zero outperforms both methods in all metrics related to persuasiveness, indicating GDP-Zero’s strong policy planning ability.\n\n3. “experiments are only conducted on a single dataset”\n     \\\n     \\\n    Please see our [Response to All Reviewers](https://openreview.net/forum?id=ZkR2bWvRpZ&noteId=hIjF1XSPou).\n\n---\n#### Response to Questions:\n1. a. “How does ‘Ask an Expert:...’ differ from this approach?”\n     \\\n     \\\n     Please see our response to *Comparisons between GDP-Zero and Existing Work* at the top.\n     \\\n     \\\n     b. “‘Ask an Expert:...’ also utilized ChatGPT for lookahead search?”\n     \\\n     \\\n    To our understanding, “Ask an Expert:...” performs planning by prompting an LLM with a fixed set of questions such as “What is the emotional status of the person?” and “What could the supporter do?”. We believe this implies that this method still relies on the inherent capability of the LLM to perform planning. To the best of our understanding, this method does not **search** for potential high-reward actions other than the response generated by the LLM. In contrast, GDP-Zero actively explores/exploits actions **other than the one proposed by the LLM**, and shows improvement compared to the LLM’s raw policy (Table 2-3). On a high level, this is achieved by GDP-Zero iteratively simulating and evaluating (L181-196) the potential outcome of various actions (including the ones not preferred by the LLM), and selecting high-reward actions to exploit/potentially high-reward actions to explore for the next tree search iteration (L168-180).\n2. “How does GDP-Zero address suboptimal policy from ChatGPT?”\n     \\\n     \\\n    On a high level, GDP-Zero (like other MCTS algorithms) utilizes the value functions estimated during the evaluation stage (L189-196) to help (e.g.) ChatGPT to avoid suboptimal action which has a low score/chance of success. Like other MCTS algorithms in general, the process of constructing a search tree, by simulating the outcome of various potential actions and finally picking the best action according to the simulations, can be seen as a single policy improvement operation (Silver et al., Nature 2017).\n\n3. “On line 155, is $a_0$ referring to a user action or a system action?”\n     \\\n     \\\n    It refers to the system action. Thank you for pointing it out! We will clarify this in our final manuscript.\n\n4. “Why is there a lack of comparison with the mentioned relevant methods in the paper?”\n     \\\n     \\\n    Please see our response to *Comparisons between GDP-Zero and Existing Work* at the top.\n\n---\n[Mastering the game of Go without human knowledge.](https://doi.org/10.1038/nature24270) (Silver et al., Nature 2017)\n\\\n[Task-Completion Dialogue Policy Learning via Monte Carlo Tree Search with Dueling Network](https://aclanthology.org/2020.emnlp-main.278) (Wang et al., EMNLP 2020)\n\\\n[Multi-Turn Target-Guided Topic Prediction with Monte Carlo Tree Search](https://aclanthology.org/2021.icon-main.39) (Yang et al., ICON 2021)\n\\\n[Seamlessly Integrating Factual Information and Social Content with Persuasive Dialogue](https://aclanthology.org/2022.aacl-main.31) (Chen et al., AACL-IJCNLP 2022)\n\\\n[Controllable Mixed-Initiative Dialogue Generation through Prompting](https://aclanthology.org/2023.acl-short.82) (Chen et al., ACL 2023)\n\\\n[Conversational Tree Search: A New Hybrid Dialog Task](https://aclanthology.org/2023.eacl-main.91) (Väth et al., EACL 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Taking the appendix into consideration, the experiments in the paper are comprehensive, providing sufficient support for the author's claim that GDP-ZERO outperforms ChatGPT in the PersuasionForGood task.", "reasons_to_reject": "a. The paper lacks citations and comparisons to similar ideas, such as  [1]\n\n[1] Zhang, Q., Naradowsky, J., & Miyao, Y. (2023). Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Models. arXiv preprint arXiv:2305.17878.\n\nb. The paper lacks essential baseline models and fails to compare them with the mentioned relevant approaches as well as the missing relevant methods. \n\nc. The experiments are only conducted on a single dataset", "questions_to_authors": "a. How does [1] differ from this approach? [1] also utilized ChatGPT for lookahead and incorporated it into task-oriented dialogue policy planning.\n\nb. Does using ChatGPT also lead to the issue of suboptimal policy, and does GDP-ZERO overlook or address this? If it addresses it, how is it resolved?\n\nc. On the third page, line 155, is a_0 referring to a user action or a system action?\n\nd. Why is there a lack of comparison with the mentioned relevant methods in the paper, and what is the rationale behind this omission?", "paper_topic_and_main_contributions": "This research introduces the GDP-ZERO approach, which leverages few-shot prompting to eliminate the necessity for model training on noisy data, and employs Open-Loop MCTS to mitigate the accumulation of simulation errors.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Zhang, Q., Naradowsky, J., & Miyao, Y. (2023). Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Models. arXiv preprint arXiv:2305.17878.", "Typos_Grammar_Style_and_Presentation_Improvements": "The presentation of lines 103-109 on the second page of the paper is quite vague."}, "extra_text": {}}, {"review_id": "s2wFgcMwJm", "reviewer": "Reviewer_e7SF", "summary": "", "questions": "A. I am not familiar with MCTS and can not understand at all the motivation behind open-loop MCTS instead of closed loop as motivated in (line 141 following). Is the open-loop MCTS something you propose? How important is it? Why did you not compare to closed-loop training which seems to be much more efficient since you don't have to re-generate histories so often.\nB. Line 147: I don't understand what you mean by saying \"we treat dialogue policy planning as stochastic MDP\". In line 134 you specify the MDP, which is also stochastic according to the transition function P? I can not assess at all the need for open-loop MCTS.\nC. Line 206: i have never seen the notation h.n and h.v before. The \".\" for me is intuitively more of an operator than \"h.n\" being a variable. The notation feels extremely awkward to me. Is this standard notation?\nD. Are any of the results in Table 3 statistically significant?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback and interest in our idea! We will add the relevant discussions in our final manuscript.\n\n\n1. “GDP-Zero requires a high amount of run time to achieve improvement”\n     \\\n     \\\n     We acknowledge this as a limitation of our current work (L331-350). However, we note that since we rely on OpenAI’s public API to access the LLMs, GDP-Zero’s runtime is greatly subject to rate limits/potential speed throttling. We believe this can be overcome by 1) engineering techniques such as having multiple OpenAI accounts, 2) requesting prioritized access to OpenAI’s LLMs, 3) other optimization techniques such as parallelizing tree search (see L343-346). We believe those techniques are tangential to GDP-Zero, and we intend to leave those optimizations to future work. On the other hand, we also expect fewer runtime concerns as research on LLM inference improves!\n\n---\n#### Response to questions:\n1. a. “What is MCTS?”\n     \\\n     \\\n     MCTS is an algorithm that can help a policy model improve its decision making. For example, it is used to achieve superhuman performance in the game of Go (Silver et al., Nature 2017). At a high level, MCTS improves an action proposed by an original policy model by building a tree that contains simulated outcomes of many potentially good actions, and picks  the final best action according to the simulation/evaluation results. Specifically, it repeats the following steps to construct such a search tree: 1) explore new actions/exploit a successful action proposed by the original model; 2) simulate the outcome of that action; 3) evaluate the quality of that action; 4) update its estimate of how good each action is.\n     \\\n     \\\n     b. “Why open-loop MCTS instead of closed-loop” MCTS?”\n     \\\n     \\\n     In the traditional closed-loop MCTS, the outcome of an action is simulated only once. In the context of a two-party dialogue, this means that it only considers one potential answer from party B, given an utterance from party A. We believe this is particularly disadvantageous and error prone for dialogue tasks, as the potential responses from party B can be diverse. We therefore utilized the open-loop MCTS variant, where the outcome of each action (i.e. party B’s response) is re-simulated/sampled every time when it is used by the tree search. We believe this added stochasticity can better capture the diverse nature of dialogue.\n     \\\n     \\\n     c. “Did you compare your algorithm against closed-loop MCTS?”\n     \\\n     \\\n     We compared our open-loop approach against the closed-loop implementation as an ablation study in Appendix C. In this ablation study, we showed that 1) the open-loop variant outperforms the closed-loop implementation, and 2) our modification to the open-loop MCTS algorithm further achieves improvement.\n     \\\n     \\\n     d. “Is the open-loop MCTS something you propose?”\n     \\\n     \\\n     No, open-loop MCTS is an already existing, general purpose algorithm. However, traditional usage of MCTS in dialogue heavily relies on training additional neural networks/user simulators to perform operations involved in the algorithm, and typically uses the closed-loop variant (Wang et al., EMNLP 2020; Yang et al., ICON 2021; see more in L87-89). To the best of our knowledge, our work is the first training-free system that utilizes MCTS to improve dialogue policy planning (Section 4). Furthermore, we showed that the open-loop MCTS along with our modifications performs better than the traditional closed-loop MCTS on the task of persuasion (Appendix C).\n\n\n2. “Why treat dialogue policy planning as a stochastic MDP”?\n     \\\n     \\\n    In a traditional (closed-loop) MCTS, the transition from one action to another is deterministic: given a dialogue context, only one response is simulated and used. Hence the transition probability from a dialogue context to that response is 100%. However, we believe this is unsuitable for dialogue as responses are diverse in reality. Therefore, GDP-Zero considers multiple potential responses per given dialogue context (the open-loop variant), and this pool of potential responses thus makes the MDP transition stochastic.\n\n\n3. Is the “notation h.n and h.v” standard?\n     \\\n     \\\n     We apologize for this confusion. We intend to use it to emphasize that the statistics (n and v) pertain to the specific history h. For clarity, we will change this notation to v_{h} and n_{h} in our final manuscript.\n\n\n4.  \"Are any of the results in Table 3 statistically significant?\"\n     \\\n     \\\n     Yes, most of the metrics in the original Table 3 (inc. donation amount, strong argument, is convincing) had p<0.1. **We conducted an additional interactive evaluation with a larger sample size to increase statistical power**, using the same procedure mentioned in the paper (Appendix H). In our new results, most results are statistically significant at the .05 level, and some have p<0.01.The updated table is presented below. \n\n\n| The chatbot                     | RAP        | ChatGPT | GDP-Zero    |\n| ------------------------------- | ---------- | ------- | ----------- |\n| donation prob. $\\uparrow$       | 0.72       | 0.73    | **0.79**    |\n| inc. donation intent $\\uparrow$ | 4.08       | 3.77    | **4.30** ** |\n| strong argument $\\uparrow$      | 3.89       | 3.91    | **4.28** *  |\n| is convincing $\\uparrow$        | 4.11       | 4.10    | **4.38** ** |\n| diverse strategy $\\uparrow$     | **3.98**   | 3.83    | 3.95        |\n| is manipulative $\\downarrow$    | 2.64       | 2.96    | **2.29** ** |\n| is natural  $\\uparrow$          | 4.25       | 4.03    | **4.38** *  |\n| is relevant $\\uparrow$          | **4.64** * | 4.31    | 4.59        |\n| is coherent $\\uparrow$          | 4.28       | 4.06    | **4.42** *  |\n\n\nwhere * denotes p < 0.05, and ** for p < 0.01. \nThe results now include 40 survey results for GDP-ZERO, 35 for ChatGPT, and 36 for RAP. We will update Table 3 and the details about population size in our final manuscript accordingly.\n\n---\n#### Typos Grammar Style And Presentation Improvements:\n1. Thank you for your feedback! We will update Figure 1 by removing some detailed text/examples and include only the necessarily high-level aspects of our algorithm.\n     \\\n     \\\n    We agree that our Appendix contained a lot of relevant information. For a holistic study, we performed many analysis and ablation experiments besides the main experiment presented in Section 4. We placed those analysis, ablation studies, example prompts, etc., in Appendix because we wanted to emphasize the GDP-Zero algorithm itself, and its improved performance compared to strong baselines such as ChatGPT and RAP. In our final manuscript, we will add short summaries/mentions of those studies in our main paper for clarity. Thank you for the suggestion!\n\n---\n\n[Mastering the game of Go without human knowledge.](https://doi.org/10.1038/nature24270) (Silver et al., Nature 2017)\n\\\n[Task-Completion Dialogue Policy Learning via Monte Carlo Tree Search with Dueling Network](https://aclanthology.org/2020.emnlp-main.278) (Wang et al., EMNLP 2020)\n\\\n[Multi-Turn Target-Guided Topic Prediction with Monte Carlo Tree Search](https://aclanthology.org/2021.icon-main.39) (Yang et al., ICON 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of policy planning by LLM prompting is interesting. Moreover, the authors provide a very detailed appendix.", "reasons_to_reject": "The authors achieve the improved performance over ChatGPT by a high amount of run time, which makes it a less feasible approach in practice. Moreover, some explanations are difficult to parse for me, please answer my questions.", "questions_to_authors": "A. I am not familiar with MCTS and can not understand at all the motivation behind open-loop MCTS instead of closed loop as motivated in (line 141 following). Is the open-loop MCTS something you propose? How important is it? Why did you not compare to closed-loop training which seems to be much more efficient since you don't have to re-generate histories so often.\nB. Line 147: I don't understand what you mean by saying \"we treat dialogue policy planning as stochastic MDP\". In line 134 you specify the MDP, which is also stochastic according to the transition function P? I can not assess at all the need for open-loop MCTS.\nC. Line 206: i have never seen the notation h.n and h.v before. The \".\" for me is intuitively more of an operator than \"h.n\" being a variable. The notation feels extremely awkward to me. Is this standard notation?\nD. Are any of the results in Table 3 statistically significant?", "paper_topic_and_main_contributions": "This paper proposes prompt-based Monte-Carlo tree search (MCTS) for goal-oriented dialogue policy planning.\nThey perform planning without training data by prompting a large language model (LLM) to act as a policy, value function, user simulator and system utterance generator. The task they consider is PersuasionForGood, where a persuader tries to convince someone to make a donation. The authors use open-loop MCTS where each tree-node is a sequence of dialogue actions without corresponding user or system utterances. The utterances need to be re-generated, which is supposed to help in circumventing compounding errors resulting from slightly improbably generated system or user responses. Results show that their approach is superior to ChatGPT in static evaluation with ChatGPT as well as interactive human evaluation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 1 is very overloaded and difficult to pass for me. While I get the gist of the paper, understanding the motivation and idea entirely is very difficult. It feels to me like this should have been a long paper instead. You provide many information in the Appendix but it should be not needed for understanding the main body."}}, {"review_id": "0qWfm6t8M8", "reviewer": "Reviewer_qWAy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We will add the clarifications in our final manuscript according to your comments!\n\n\n1. “experiments are only conducted on a single dataset”\n     \\\n     \\\n     Please see our [Response to All Reviewers](https://openreview.net/forum?id=ZkR2bWvRpZ&noteId=hIjF1XSPou).\n\n2. “There is no comparison with other baselines, such as traditional non-LLM state-of-the-art methods”\n     \\\n     \\\n     To date, training-free methods similar to GDP-Zero have not been used for policy planning. Prior work that utilizes tree search on dialogue relies heavily on model training (Wang et al., EMNLP 2020; Yang et al., ICON 2021; Väth et al., EACL 2023; see more in L87-89). For a fair evaluation, we compared this training-free approach with two strong baselines in our experiments: directly prompting ChatGPT (training-free) and using the RAP system (training a generator with a scripted planner). ChatGPT had been famous for achieving strong performance in a wide-range of dialogue tasks, and to our knowledge, **RAP is the current state-of-the-art on Persuasion4Good** (Chen et al., AACL-IJCNLP 2022, Chen et al., ACL 2023). GDP-Zero surpassed both ChatGPT-only and RAP while being a training-free approach.\n\n---\n#### Typos Grammar Style And Presentation Improvements:\n1. Thank you for pointing this out! We will correct all “dialog” to “dialogue” in our final manuscript.\n2. The metric “is manipulative” is better if it is lower. We indicated this with “is manipulative ↓”. We will add additional clarification in the table caption in our final manuscript.\n3. Thank you for spotting this! We will update our equations with punctuations in our final manuscript.\n---\n\n[Task-Completion Dialogue Policy Learning via Monte Carlo Tree Search with Dueling Network](https://aclanthology.org/2020.emnlp-main.278) (Wang et al., EMNLP 2020)\n\\\n[Multi-Turn Target-Guided Topic Prediction with Monte Carlo Tree Search](https://aclanthology.org/2021.icon-main.39) (Yang et al., ICON 2021)\n\\\n[Seamlessly Integrating Factual Information and Social Content with Persuasive Dialogue](https://aclanthology.org/2022.aacl-main.31) (Chen et al., AACL-IJCNLP 2022)\n\\\n[Conversational Tree Search: A New Hybrid Dialog Task](https://aclanthology.org/2023.eacl-main.91) (Väth et al., EACL 2023)\n\\\n[Controllable Mixed-Initiative Dialogue Generation through Prompting](https://aclanthology.org/2023.acl-short.82) (Chen et al., ACL 2023)\n\\\n[Conversational Tree Search: A New Hybrid Dialog Task](https://aclanthology.org/2023.eacl-main.91) (Väth et al., EACL 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a method for using Open-Loop MCTS to plan dialogue policies without training large models, which is practical. \n\nThe Appendix section provides sufficient details to ensure the reproducibility of the proposed method.", "reasons_to_reject": "Please refer to the above \"Paper Topic And Main Contributions\" part.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is based on ChatGPT and uses Open-Loop MCTS for dialogue policy planning. Overall, I think it is a decent piece of work. However, there are some issues:\n\n1、 The experiments are only conducted on a single dataset\n\n2、 There is no comparison with other baselines, such as traditional non-LLM state-of-the-art methods\n\nThe above two shortcomings make it less convincing outcomes.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Some typos include:\n\nThe spelling of \"dialog\" is inconsistent; in some places, it is written as \"dialog,\" while in others, it is written as \"dialogue.\"\n\nTable 3 is manipulative, highlighting scores that are not the highest.\n\nPunctuation is missing after the equations, making the sentences incomplete."}}]

Author rebuttal:

Meta-review: This paper presents a novel approach to policy planning for persuasion dialogue using Open-Loop MCTS without any model training.  Experiments on the PersuasionForGood dataset as well as an interactive human evaluation demonstrate that the proposed GDP-ZERO model outperforms ChatGPT.  The reviewers agree this is an interesting new approach to persuasion dialogue and appreciate the extensive experiments and results, further detailed in an elaborate appendix.  Some concerns were raised about a lack of comparisons with other methods in the experiments, which were subsequently addressed in the rebuttal, largely to the reviewers' satisfaction.  The other main concern raised was the fact that evaluations were carried out on one dataset only.  In response, the authors pointed practical limitations of evaluations in other datasets/domains, because these require hypothetical scenarios to be followed by the subjects.  This is understandable, but also leaves us with the question to what extent the proposed approach is applicable to and effective in other genres of task-oriented dialogue, which is what the title seems to suggest.
Decision: Accept-Main