Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration
Authors: Yang Deng
Lizi Liao
Liang CHEN
Hongru WANG
Wenqiang Lei
Tat-Seng Chua
Venue: EMNLP 2023 Findings (2023)
Keywords: Proactive Dialogue
Asking Clarification Question
Target-guided Conversation

Abstract: Conversational systems based on Large Language Models (LLMs), such as ChatGPT, show exceptional proficiency in context understanding and response generation. However, they still possess limitations, such as failing to ask clarifying questions to ambiguous queries or refuse users' unreasonable requests, both of which are considered as key aspects of a conversational agent's proactivity. This raises the question of whether LLM-based conversational systems are equipped to handle proactive dialogue problems. In this work, we conduct a comprehensive analysis of LLM-based conversational systems, specifically focusing on three key aspects of proactive dialogues: clarification, target-guided, and non-collaborative dialogues. To trigger the proactivity of LLMs, we propose the Proactive Chain-of-Thought prompting scheme, which augments LLMs with the goal planning capability over descriptive reasoning chains. Empirical findings are discussed to promote  future studies on LLM-based proactive dialogue systems.

Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided and Non-collaboration
Yang Deng', Lizi Liao", Liang Chen', Hongru Wang', Wenqiang Lei', Tat-Seng Chua'
National University of Singapore SSingapore Management University
The Chinese University of Hong Kong 'Sichuan University
[ydeng, dcsccs Jenus. edu. sg, Izliao@smu. edu. edu. sg
{ (lchen, hrwang] cuhk. edu. hk, wengianglei@gman com
Abstract
showledge-groundde dialogues (Bang et al., 2023), task-oriented dialogues (Zhang et al., 2023), and emotion-awar dialogues (Zhao et al., 2023). Despite the strength of ChatGPT, there are still several limitations', such as failing to ask clarifica- tion questions to ambiguous user queries refuse problematic user requests. These kinds of capabili- ties are typically regarded as the proactivity of the conversational system (Deng et al., 2023b), where the system can create or control the conversation to achieve the conversational goals by taking ini- tiative and anticipating impacts on themselves or the human users. Thus, it raises the question: Are these LLM-based conversational systems equipped to manage proactive dialogue problems?
Conversational systems based on Large Lan- guage Models (LLMs), such as ChatGPT, show exceptional proficiency context understand- ing and response generation. However, they still possess limitations, such as failing to ask clarifying questions to ambiguous queries or refuse users' unreasonable requests, both of which are considered as key aspects of a con- versational agent's proactivity. This raises the question of whether LLM-based conversational systems are equipped to handle proactive dia- logue problems. In this work, conduct comprehensive amalysis of L.L--hased conver- sational systems, specifically focusing on three key aspects of proactive dialogues: clarifica- tion, target-guided, and non-collaborative di- alogues. To trigger the proactivity of LLMs, we propose the Proactive Chain-of-Thoup prompting scheme, which augments LLMs with the goal planning capability descrip- tive reasoning chains. Empirical findings are discussed to promote future studies on L.MM- based proactive dialogue systems.
In this work, conduct the first comprehen- sive analysis of LLM-based conversational systems on three common aspects of proactive dialogues, including 1) clarification in information-ssekking dialogues (Guo et al., 2021; Deng et al., 2022a) where the system is required to proactively ask clar- ification questions when encountering ambiguity in user queries; 2) target-guided open-domain dia- logues (Tang et al., 2019; Wu et al., 2019) where the system is required to proactively lead the con- versation towards the designated target; and 3) non-collaborative task-oriented dialogues (Li al., 2020; Zhou et al., 2020; Deng et al., 2023a) where the system and the user do not share the same con- versational goal while the system aims to strategi- cally reach consensus with the user.
1 Introduction
Conversational systems are envisioned to provide social support or functional service to human users via natural language interactions. Most research typically centers around system's re- sponse capabilities, such as understanding the dialogue context (Wu et al., 2020; Chen et al., 2022: Deng et al., 2022b) and generating appro- priate responses (Zhang et al., 2020b; Roller et al., 2021). The popularity of conversational systems has grown unprecedentedly with the advent of Chat- GPT, which showcases exceptional capabilities of context understanding and response generation with large language models (LLMs). Recent stud- ies observe that, compared with current fine-tuned state-of-the-arr (SOTA) methods, ChatGPT can still achieve competitive performance under zero-shot setting on different dialogue problems, such as the
Motivated by the emergent capabilities of LLMs (Wei et al., 2022a,b) reasoning over texts, some recent studies investigate in-context learning chain-of-thouge prompting schemes on planning (Huang et al., 2022) or taking actions (Yao et al., 2022) interactive environments. Similarly, strat- egy learning and goal planning attach great impor- tance in proactive dialogue systems. In order to enhance the proactivity of LLM-based conversa-
la stated in https: http:///opppaiiccac wwt.b/oog/chagg......oooo

---

(3) Cenficcn bidlggan Abg-
(23 collntont bidogam: Crapsimkagga
Bockgroadd
book, Argle
dreang
Task Rockyreadd: The how indoriationss is "Soling my 2006 Taystt 4 Russor wit only 1264 orijiss wiled The trais is grant tondition unt no machoniod flovr whittoener and citees occient simaary... don't
Mestery
ind blue chan, She draw pictans her trather, Her nather found tock. wax baok.
mazlic your time and my hime I'm bay pay" The acting proce as 14500. The target prcce 8700.
Converaction Hastryy ["Aayer:: thane frae ateat you give me 50007" SSller" "Thtt way too lttll
"Rayer" How
Cotmorcation Histwyy [Usett: "Wort di the drea? "Syttee" "wer nuther" "Moer"
'What did her nother finer; "syster" "The sook", "Uber" "Whet color war I?)
Standardd
Standard
0i0 Frouctive
Prosytt Gieen the Tide bockground and the
Stratsgy: ("Oudé repport" "Show dera "Negetive acntirest" _]]
Aroept: Given beckgroed end samarasrien lan hartery, pleass the
Acti l'Oinectly Answer" "
conversition histery, piesss generate the raapowes Resperse: How abeot ne neet is the ricda at 100007
clarficction Querhert unf]
affer". badkgrunn the converaption histery, first andyns the curment regetictiin pregrets and candi fer apprppicte regatistion geel and then select appnepient stratigg ea sctions to genenate the resporact Requoree: The callor timm y rejecte the price. 21 would be bether to propase highar price Striley Negetive
"Prspsingg ppection"
Recperse: Gress
rapprae
X
Pronge:
badigrcued
first
whather cuestion
(26) Prosciiee
curment turn
Stretegy: ('Bals repport: "Show demvnance"
and tiss colect oppr spiate octiss gener she the reeparss
Act (jsk e quertion" "Exest offer" --)
L'Diractly Ansarr TAsk corification Quaurion on']
Prueeyt
bockgroush
Resporne: There are two books that Angln's is mother fourd 31 unclerr
approamate
Arcegtt Given the tadk bock bockggoudd end the comeraction on hartony please las approprittc
which base refermed Therefons.
Gesporas: The strehegy la Conmunittte palitely
actione to geasnene the repporer:
question onaiiga
The ocl is Propoding counter price. intersters
Iroposing
Resporse: Ank clrnitiation quaction Could you prewide more (immsttion?
X
How obout we noot ha lalaaay 102507 what do you thoi??
BL SL-rate:0 7.74
poy 8200 let
gi. retii 1.00
Figure I: Examples of three kinds of prompting schemes for proactive dialogues. In the example of non-collaborative dialogue, the system plays the role of "Buyer", and the sale-to-list (SL) ratio shows the effectiveness of negotiation, which is calculated by (listed price bargain price)/ (listed price- buyer target price). The higher ratio means the current bargain price is closer to the target.
tional systems, we design the proactive chain-of- thought prompting (ProCoT) scheme. As shown in Figure 1, with standard prompting, LLM-based sys- answer
the initiative towards the conversational goal.
Specifically, the main findings of the evaluation of LLM-based dialogue systems include: 1) They barely ask clarification questions when encoun- tering ambiguous queries, and ProCoT largely overcomes this issue, though the performance is still unsatisfactory in domain-specifie applica- tions (54.1). 2) They are proficient at performing topic shifting towards the designated target, but tend to make aggressive topic transition. ProCoT further improves this capability by planning a smoother transition ($4.2). 3) They fail to make strategic decision and tend to compromise with the opponent. The key challenge is how to effee- tively optimize the strategy learning (61.)).
tems directly provide a randomly- to the ambiguous user question (1a), or generate a general bargain response without any negotiation strategy (2a). When providing the system with options to take different dialogue acts (proactive prompting), the generated responses unaware of the conversational goal, such as generating under- specified clarification questions (1b) and conserva- tive negotiation responses (2b). To this end, Pro- CoT first instructs the system to generate descrip- tive thoughts about intermediate steps of reasoning and planning for reaching the conversational goal, and then make the decision of the next action to take. Finally, the system generates an appropriate response based on the decided action (Ic & 2c).
2 Related Works
We conduct extensive experiments with two LLM-based conversational systems, including ChatGPT and an open-sourced model, Vicuna (Chi- ang et al., 2023). With the aforementioned three types of prompting schemes, we compare these LLM-basedd conversational systems with fine-tuned SOTA dialogue models. The main contributions of this work can be summarized as follows:
Proactive Dialogues. Recent years have wit- nessed many advanced designs on developing proactive dialogue systems (Liao al., 2023) for various applications. For example, target-guided dialogues aim to proactively lead the conversa- tion to either a designated target topic (Tang et al., 2019) or a pre-defined knowledge entity (Wu et al., 2019). Existing studies typically adopt keyword transition (Qin et al., 2020; Zhong et al., 2021) or knowledge graph reasoning (Yang et al., 2022; Lei et al., 2022) techniques to proactively plan the topic thread towards the target. Besides, in information-seeking dialogues, proactive dialogue systems can ask clarification questions for clari- fying the ambiguity of the query or question in conversational search (A (Aliannejadi et al., 2021) and
This work presents the first comprehensive eval- uation on the proactivity of LLM-based dialogue systems, including the handling of clarification, target-guided, and non-collaborative dialogues.
We design the proactive chain-of-thouge prompt- ing scheme to endow LLM-based dialogue sys- tems with the capability of planning and taking

---

question answering (Guo et al., 2021; Deng al.. 2022a). In addition, under the non-collaborative setting, the system and the user have competing goals towards the task completion but the system aims to proactively reach an agreement favorable to itself (Zhou et al., 2020), such as negotiating a product price (He et al., 2018) or persuading users to make donation (Wang et al., 2019).
Given the task background DD and the conversa- tion history C, instruct the LLM to generate the response r. specific, the task background can be the grounded document in clarification dialogues or the target description in target-guided dialogues.
Proactive Prompting. Proactive prompting aims to provide alternative options for LLMs to decide what kinds of actions should be taken in the sponse, instead of simply responding to the instruc- tion. can be formulated as:
Large Language Models for Dialogues. Previ- ous dialogue systems, such as DialoGPT (Zhang et al., 2020b), Meena (Adiwardana et al., 2020), BlenderBot (Roller et al., 2021), LaMDA (Thop- pilan et al., 2022), typically fine-tune pre-trained language models on public dialogue data. Inspired by the success of ChatGPT, recent practices build dialogue systems through conducting supervised fine-tuning on open-source large language models, such as LLaMA (Touvron et al., 2023), with either constructed serrcctionfollooinng examples (e.g., Alpaca (Taori et al., 2023)) or distilled conversation data (e.g., Vieuna (Chiang al., 2023)) from Chat- GPT. As all these LLMM-based dialogue systems trained follow the user's instruction, it remains a question on whether these systems can take the initiative for handling proactive dialogues.
p(a, (D,,,,
(2)
Given the task background D, the conversation history C, and set of possible dialogue acts A, instruct the LLM select the most appropriate di- alogue act A and then generate the response r. For example, the dialogue act can be Ask a Clarifi- cation Question or Directly Answer the Question in clarification dialogues, different negotiation strate- gies in non-collaborative dialogues, or different conversation topics in arget-guided dialogues.
Proactive Chain-of-Thougll Prompting. In or- der to endow LLMS with the capability planning and taking the initiative towards the ultimate goal, we develop the proactive th-ouuhh prompt- ing scheme ProCo. It involves the analysis of the next action to take by performing dynamic rea- soning and planning for reaching the conversational goal. ProCoT can be formulated as:
Prompting in Dialogue Systems. To induce knowledge from LLMs, various prompting meth- ods are designed for zero-shot or few-shot learn- ing in dialogue applications, such as task-oriented dialogues (Lee et al., 2021; Mi et al., 2022), moowledge-grounder dialogues (Shuster et al.. 2022; Liu et al., 2022; Wang et al., 2023c), and open-domain dialogues (Chen al.. 2023b; Lee al., 2023; Wang al., 2023a). For example, Chen al. (2023c) propose to prompt LLMs for controllable response generation in emotional sup- port and persuasion dialogues, conditioned on the ground-truth dialogue strategies. In this work, we aim at prompting LLMs to proactively interact with the users.
p(t, a, C...
where t is the thought description for the decision- making process of the next action. For example, in clarification dialogues, can be the ambiguity anal- ysis of the current user question as in Figure 1(cc). While in collabbbabor dialogues, t can be the goal completion analysis of the current negotiation progress as Figure 1(2c).
Evaluation
3 Prompting LLMs to be Proactive
We evaluate the proactivity of LLM-based conver- sational systems from three perspectives, includ- ing the capability of asking clarification questions (6 4.1), guiding the conversation towards the des- ignated target (6 4.2), and strategically handling conflicting goals (6 4.3).
As presented in Figure 1, we describe the prompt- ing schemes, including the standard, proactive, and proactive chain-of-thought (ProCoT) prompting.
Standard Prompting. In order to instruct LLMs to perform specific dialogue tasks, the typical prompting scheme can be formulated
4.1 Clarification Dialogues
Clarification in information information-sccking dialogues (Za- mani et al., 2022) refers to the process of seeking
P(r(D.C).
(1)

---

Abg-CoQA
PACIFIC
1 and ROUGE-2 (F1) for the evaluation of CQG. In
CNP
CQG
CIP
CQG
addition, since the automatic lcxical matching met-
Method
Shot
Procupt
Fl
BLEU-1
Help.
Fl
ROUGE-2 2 Delp
rics may fail to actually estimate the clarification ca-
Buxeline
22.1
36.5
30.0
79.0
69.2
38.2
pability of the generated clarifying questions (Guo
SOTA
23.6
38.2
56.0
86.9
90.7
80.1
Standard
11.3
0.0
et al., 2021), we also adopt human evaluation to
Standard
11.4
0.0
2.5
0.0
score whether the generated question is helpful for clarifying the existing ambiguity (Help.).
0
Prouctive 4.1
13.2
0.0
2.3
2.3
0.0
Vicuna 13B
Proactive 12.1
13.2
4.5
0.0
3.3
0.0
0
ProCoT 1.4
21.3
9.1
9.7
3.8
10.5
ProCoT 18.3
23.7
22.7
27.0
41.3
33.1
Usage of LLMs. To facilitate reproducibility, we adopt a static version of ChatGPT, i.e., 5-tubbo-300 and set the temperature to for generating the deterministic outputs with the same inputs. In addition, we adopt an open-source LLM, i..., Vicuna-13B- 10- ta-v 1, for the eval- uation. The maximum number of new tokens is set to 128 for the generation.
0
Standard
12.1
0.0
1
Standard
12.3
0.0
2.0
0.0
Pronetive 22.0
13.7
17.6
19.4
29
CharGPT
Pronctive 20.4
23.4
23.5
17.7
14.0
12.5
0 ProCoT 23.8
21.6
32.4
28.0
21.5
26.7
ProCoT 27.9
18.4
45.9
27.7
162
Table I: Experimental results Abg-CoQA and PA- CIFIC datasets, whose baseline and SOTA results are adopting from Guo al. (2021) and Deng et al. (2022a). Bold and underlined results denote the best performance for each L.M and the fine-tuned methods, respectively.
Prompting Schemes. We evaluate the three prompting schemes introduced Section 3, includ- ing standard, proactive, and ProCoT prompting. In addition, we report their results under both zero- shot and few-shot settings. Due to the limitation of the maximum sequence length in Vicuna (2,048 tokens), only apply one-shot in-context learning for comparisons. The complete prompts adopted for evaluation is presented in Appendix C.
further information or details to better understand the topic or question hand. In this context, clar- ification is an important part of the dialogue as helps to ensure that the information being shared is accurate and complete.
4.1.1 Problem Definition
Following previous studies (Aliannejadi et al., 2021: Guo et al. 2021; Deng et al., 2022a), the problem of asking clarification questions can be decomposed into two subtasks: 1) Clarification Need Prediction (CNP) to identify the necessity of clarification in the current turn, and 2) Clarifi- cation Question Generation (CQG) to produce an appropriate clarifying question if needed. Given the grounded document D and the dialogue con- text C = {1, .1..... ..... 4t 1, at 1: [t],, the dialogue system aims to first predict the binary ambiguity label y on whether the current question qi needs be clarified. If so, a corresponding clarification question should be generated as the response ot for clarifying the ambiguity.
4.1.3 Experimental Results
Table summarizes the evaluation results on Abg- CoQA and PACIFIC datasets. There are several notable observations as follows:
LLM-based conversational systems fail to ask clarification questions. Under standard prompt- ing, both Vicuna and ChatGPT fail to ask clari- fication questions when encountering ambiguous queries, according to the human evaluation on the helpfulness (Help.) of the generated responses for clarifying ambiguity. Even with one-shot demon- stration, in-context learning (ICL) still cannot pro- vide them with such ability. Under proactive prompting, given the option of clarification, Vi- cuna's ability to accurately take this action is still quite limited, with the F1 scores close to 0. In contrast, CharGPT becomes capable of asking clar- ification questions on Abg-CoQA, as evidenced by the improvement on both F1 and Help. scores.
4.1.2 Experimental Setups
Datasets. We evaluate the capability of asking clarification questions in LLM-based dialogue sys- tems two types of datasets: 1) Abg-CoQA (Guo et al., 2021) in general domain, and 2) PA- CIFIC (Deng et al., 2022a) in finance domain. De- tails on these datasets can be found in Appendix A.
ProCoT effectively endows LLM-based conver- sational systems with the capability of asking clarification questions. Zero-shot ProCoT is not working in Vicuna, but one-shot ICL can largely improve the performance. As for Abg-CoQA, Chat- GPT with zero-shot ProCoT achieves competitive
Evaluation Metrics. Following previous stud- ies (Guo et al., 2021; Deng et al., 2022a), we use the Fl score for the evaluation of CNP, and BLEU-

---

Abg-CoQA
PACIFIC
get (Tang al., 2019). According to different appli- cations, the target can be topical keywords (Zhong et al., 2021), knowledge entities (Wu et al., 2019), or items to be recommended (Deng et al., 2023c)-
Wrong Aspect
21%
30%
Under-spec. Clari.
16%
23%
Over-spec, Clari.
15%
5%
Generation Error
48%
42%
4.2.1 Problem Definition
Table 2: Statistics of error analysis.
Given a target D that is only presented to the agent but unknown the user, the dialogue starts from an arbitrary initial topic, and the system needs produce multiple turns of responses (un) to lead the conversation towards the target in the end. The produced responses should satisfy (i) transition smoothness, natural and appropriate content under the given dialogue context, and (ii) target achieve- ment, driving the conversation towards the desig- nated target. The problem is typically decomposed into two subtasks (Tang et al.. 2019; Zhong et al., 2021; Yang et al., 2022): next topic selection and transition response generation.
performance with SOTA fine-tuned methods on the CNP task (F1). but the generated clarification questions are still unsatisfattory (Help.). One-shot ICL further improves the performance of ChatGPT with ProCoT to a great extent. The case study in Appendix D.I shows that ProCoT also improves the explanability of asking clarification questions.
As for domain-specific problem, there is still a noticeable gap from the fine-tuned methods. Although ProCoT has already largely enhanced the capability of asking clarífication questions, the per- formance of LLMs on the -eeaai--pecific task, i.e., PACIFIC (Finance),, is still far behind the fine- tuned methods. In fact, with fine-tuning on domain- specific data, the SOTA method can achieve re- markable performance on PACIFIC, i.e., 86.9 (F1) for CNP and 80.1 (Help.) for CQG, indicating the importance of domain knowledge.
4.2.2 Experimental Setups
Datasets. We first conduct turn-level evaluation of the target-guided capability a next-turn target- oriented dataset OTTers (Sevegnani al., 2021), which requires the dialogue system to proactively bridge the current conversation topic to approach the target. Furthermore, we adopt TGConv (Yang et al., 2022) to testify the ability to guide the multi-turn conversation to the target topic as the dialogue-level evaluation. Details can be found in Appendix A.
4.1.4 Error Analysis
In order to find out the reason why LLM-based dialogue systems with ProCoT prompting fall short of handling domain-specific clarification dialogues, we randomly sample 100 error cases in clarification question generation from each dataset for analy- sis (all cases are generated by ChatGPT with one- shot ProCoT). We categorize these failure cases into four groups, including Wrong Aspect, Under- specified Clarification, Over-specified Clarifica- tion, and Generation Error. The details and ex- amples can be found in the Appendix B. The statis- ties of error analysis is presented in Table 2. can be observed that the proportion of failure cases attribute to the wrong aspect and under-specified clarification in PACIFIC (Finance) is higher than that in Abg-CoQA (General). This indicates that ChatGPT may lack of certain domain knowl- edge required for asking precise and specific clarification questions.
Automatic Evaluation Metries. Following pre- vious studies (Sevegnani et al., 2021; Yang et al., 2022), we adopt the hits@k (k: E [1,3) for evaluat- ing next topic prediction. Three text generation metrics, including BLEU, ROUGE-L, and ME- TEOR scores, are used for the evaluation of re- sponse generation on the OTTers dataset.
As for the dialogue-Ievel evaluation on the TG- Conv dataset, we follow existing studies (Yang et al., 2022; Wang et al., 2023b) to simulate multi- turn conversations via self-play (Tang et al., 2019), where the simulated user is unaware of the tar- get topic. Three aspects are evaluated: 1) Suec. is the success rate of generating the target word within 8 turns of conversations; 2) Turns is the average turns of all dialogues that successfully reach the target word; and 3) Coh. is the contex- tual semantic similarity between the last utterance and the generated response, which is measured by MiniLM (Wang et al., 2020).
4.2 Target-guided Dialogues
Instead of making consistent responses the user- oriented topics, the dialogue system for target- guided dialogues is required to proactively lead the conversation topics towards a designated tar-

---

Response Generation
Next Topic Prediction
models perform closely to fine-tuned methods in terms of lexical similarity with the reference re- sponse. 3) One-shot ICL casts no positive impact on the performance and may even lead to worse results in next-topic prediction. This indicates that it is difficult for LLMs to enhance the topic shifting capability from limited demonstrations.
Mothod
Prompt
BLEU
METEOR
hits@1
hise@3
GPT2
11.38
10.26
17.67
4.39
15.79
DKRN
12.86
11.90
17.92
CKC
13.34
11.65
14.77
6.87
21.89
TopkG
15.35
13.41
22.16
2,78
22.06
Stundard 10.01
13.27
16.00
12.01
19.05
Stundadd
10.63
14.81
17.53
12.10
16.13
Prouctive
1.41
18.45
15.45
9.41
19.89
Vicuna-13B
Pronctive
13.87
20.96
21.36
12.90
22.31
ProCaT
5.27
16.59
11.56
ProCaT
13.38
19.70
20.62
15.05
20.70
Only ProCoT prompting with one-shot demon- strations can improve the topic shifting capabil- ity. Without demonstrations, proactive and Pro- CoT prompts perform even worse than standard prompts, since LLMs may confuse about what kinds of topics are desired. For example, we ob- serve a typical mistake that LLMs tend to analyse the next topics using questions, such as "What kind of food do you like?", leading to narrow topic for the next turn. With one-shot demonstrations, ChatGPT with proactive prompts continues to un- derperform compared standard prompts when comes to accurately predicting suitable topics towards the target. However, is worth noting that only ProCoT prompts consistently show an improvement in the performance of all LLMs for next topic prediction.
Standard 11.34
20.62
13.44
Stundaed 14.41
19.29
17.73
15.86
26.34
CharGiP
Prozctive 14.09
21.06
7.53
Pronetive 14.74
19.59
16.29
8.60
21.25
ProCaT 10.20
19.57
15.97
12.63
23.92
ProCoT 9.63
19.82
17.19
17.74
29.57
Table 3: Turn-level evaluation results on Next Topic Prediction and Transition Response Generation.
Human Evaluation Metrics. We also conduct the same human evaluation as Yang et al. (2022), including two dialogue-level metrics with the fol- lowing instructions provided for annotators:
Global-Coherenn (G-Coh.): Whether the entire dialogue is logically and topically coherent.
Effectiveness (Effect.): How efficiently the target is achieved.
A total of 100 dialogues are generated through sim- ulation for each method. Three annotators assign ratings to the generated dialogues on scale of [O, 1, 21, where higher scores indicate better quality.
4.2.4 Dialogue-Ievel Evaluation
Table 4 shows the dialogue-Ieve evaluation results on TGConv. We draw the following conclusions:
LLM-based dialogue systems tend to make ag- gressive topic transition. The results demon- strate the effectiveness of LLMs in steering the conversation towards the designated target, with ChatGPT exhibiting nearly perfect success rates (Suce.). Compared with baselines, LLMs also ex- cel in generating more coherent responses that align with the dialogue context (Coh.), showcasing their impressive abilities in context understanding and response generation. Furthermore, the analysis re- veals that ChatGPT basically achieves the target topics within just three turns, suggesting its ten- dency to generate responses that aggressively in- volve the desired topic. Similar observations can be made with Vicuna using standard prompting.
Baselines. We report the results of several fine- tuned baselines for target-guided dialogues, includ- ing GPT-2 (Radford et al., 2019), DKRN (Qin et al., 2020), CKC (Zhong et al., 2021), TopKG (Yang et al., 2022), and COLOR (Wang et al., 2023b).
4.2.3 Turn-level Evaluation
Table 3 shows the turn-level evaluation results on OTTers. There are several notable observations:
LLM-based dialogue systems are proficient at performing topic shifting towards the desig- nated target. According the performance of LLMs with standard prompting, we observe that: I) As for the next-topic prediction (hits@k) k), thanks to the extensive knowledge across various topics, zero-shot LLMS can achieve competitive (Vicuna) or even better (ChatGPT) performance than the fine-tuned methods. 2) As for the transition re- sponse generation, automatic evaluation metrics (BLEU, METEOR, R-L)² suggest that zeoo
ProCoT prompting enables smoother topic transition of target-guided dialogues. Under proactive prompting, the response coherency is im- proved by the topic planning. However, the suc- cess rate is negatively affected, which attributes to
can be described in different ways rather than the reference response. We mainly discuss the topic shifting capability in terms of the performance on next topic prediction.
NNoe that the automatic evaluation of response generation is less reliable (Sevegnani et al., 2021), as the same topic

---

Easy Target
Hard Target
Nego. Strategy
Dial. Act
Resp. Gen.
Method
Shot Prompt
Succ.(%)
Turns
Cob.
Succ.(5)
Turns Cob.
Method
Shot Prompt
F1
ACC
FL
AUC
BLEU
BERTScore
GPT2
22.3
2.86
0.23
17.3
2.94
0.21
FeHED
17.6
55.8
20.6
76.9
23.7
27.0
DKRN
38.6
4.24
0.33
21.7
7.19
0.31
HEDARNN
23.2
65.3
33.0
83.1
22,5
22.8
CKC
41,9
4,08
0.35
24.8
6.88
0.33
HED+TFM
26.3
68.2
32,5
24.4
27.7
TopKG
48.9
3.95
0.31
27.3
4.96
0.33
DIALOGRAPH
26.1
68.1
314
85.6
242
28.1
COLOR
66.3
0.36
20.1
0.35
Stundard
1.7
-14.0
Stundard
1.9
-2.8
Standard
63.0
2.63
0.43
62.5
2.45
0.39
Prouctive 20,6
2.3
Staddrd
Vicuna- 131
Proxctive 15.2
50.0
6.7
50.8
2.6
-0.9
Vicuna- 13B
0
Proactive
37.8
2.71
0.48
35.6
2.56
0.55
ProCeT 19.0
Prouctive
48.3
2.71
0.50
34,6
2.95
0.51
ProCoT 17.8
48.9
7,7
52.5
2.6
-0,9
ProCoT
4.22
4.17
0.45
Suadard
ProCoT
72.3
3.55
0.52
59.8
3.81
0.48
Standard
3.1
0.7
0
Standard
97.5
2.26
0.38
96.3
2.30
0.41
ChaGGTT
Prosctive 128
51.5
13.3
4.2
Stundadd
96.3
2,42
2.28
Prouctive 13.7
50.9
12.00
54.9
3.9
2.9
CharPT 0 Proactive
85.9
3.20
01.47
83.0
2.83
0.43
PruCoT 10.8
50.4
10.1
54.
3.7
Preactive
90.7
2.86
0.36
86.2
2.94
0.31
ProCoT 15.1
53.5
16.3
58.2
3.9
1.5
ProCoT
96.3
2,47
2.29
ProCof
95.9
2,63
0.45
92,1
2,47
0.39
Table 6: Evaluation results Negotiation Strategy Pre-
diction, Dialogue Act Prediction, and Response Genera- tion.
Table 4: Dialogue-Leve evaluation results on target-
guided dialogues.
4.3 Non-collaborative Dialogues
Easy Target
Hard Target
Method
Prompt
G.Coh. Edffect.
G. Coh. Liffect,
Unlike collaborative task-oriented dialogue set- tings (Zhang al., 2020c), where the user and the system work together reach a common goal (e.g., booking hotels), non-collaborative dialogues, the user and the system have a conflict of interest but aim to strategically communicate to reach an agrec- ment (e.g., negotiation) (Zhan et al., 2022). The system is required to leverage series of proactive strategies to reach an agreement favorable to itself, instead of passively following the user's intents.
TopKG
1.42
1.24
1.21
1.10
Standard
1.37
1.60
1.30
1.49
Vicuma-138 13B
Pronctive
1.51
1.27
1.26
1.23
ProCaT
1.57
1.70
1,38
1.59
Staddadd
0.97
1.92
0.84
1.89
CharGPT
Proserive
1.24
1.77
1.12
1.68
ProCoT
1.20
1,90
1.14
1.85
Table 5:
Huraa evaluation on target-guided dialogues. All reported methods are under the one-shot setting,
its drawback of next topic prediction discussed in Section 4.2.3. Under ProCoT prompting, Vicuna effectively guide the conversation towards the des- ignated target with a smoother (higher Coh.) and more engaging (higher Turns) conversation than using standard prompting. However, it still remains challenging for ChatGPT to perform a smooth topic transition. Case studies in Appendix D.2 provide intuitive examples for illustrating these observa- tions.
4.3.1 Problem Definition
Given the dialogue history C = (u1, ...; 11-1) and the dialogue background D, the goal to generate a response uy with appropriate dialogue strategy a that can lead to a consensus between the system and user. A set of dialogue strategies A is pre-defined for prediction. Based on different applications, the dialogue strategy can be coarse dialogue act labels or fine- -raiiga strategy labels. The dialogue back- ground includes the system's goal and the related grounded information, such as item descriptions in bargain negotiation (He et al., 2018) and user profile in persuasion dialogues (Wang et al., 2019).
4.2.5 Human Evaluation
Table 5 presents the human evaluation results on TGConv. Compared with TopKG, LLMs demon- strate remarkable efficiency in achieving the des- ignated target (Effect.). However, the global co- herence (G-Coh.) of the generated dialogues by ChatGPT is quite low, which may harm the user en- gagement and experience during the conversation. Thus, the proficiency of controllable generation in LLMs double-edged sword for target-guided dialoges.. The key challenge of LLLMs is how to guarantee the topical smoothness and coherence of the generated transition responses.
4.3.2 Experimental Setups
Datasets. We use the CraigslistBargain dataset (He et al., 2018) for evaluating the capability of strategically handling non-collaboration in LLM- based dialogue systems. The dataset was created under the bargain negotiation setting where the buyer and the seller are negotiating the price of an item on sale. Details can be found in Appendix A.

---

Metric
Standard
Proactive ProCoT
Gold
Persuasive: Whether the seller is persuasive in bargaining the price.
Persuasive
1.24
1.28
1.43
1.54
Cohereat
1.56
1.66
1.74
1.69
Natural
1.94
1.82
1.89
1.97
Coherent:
Whether the seller's responses are on topic and in line with the conversation history.
Win Rates
Standard
0.22
0.24
0.42
-v.. Proactive
0.25
0.31
0.45
ProCoT
0.20
0.18
0.34
Natural: Whether the seller is human-like -ike.
Gold
0.19
0.09
0.23
Sale-to-Lisst iist Ratio
0.48
0.43
0.54
0.64
Bargain Price: What is the current bargain price from the seller's side.
Table 7: Human evaluation on con-collaborative dia-
logues. reported methods are based on ChatGPT under the one-shot setting. Gold denotes that in- struct the LLMs to generate responses conditioned on the reference dialogue acts and negotiation strategies.
Win: Assume you are the seller. Which dialogue system you would like to use for bargain the price with the buyer WWin/Tie/Lose).
Automatic Evaluation Metrics. Following the previous study (Joshi et al., 2021), conduct comprehensive evaluation over three subtasks, in- cluding negotiation strategy prediction, dialogue act prediction, and response generation. We report the Fl and ROC AUC scores for strategy predic- tion and dialogue act prediction, where the former one is a multi-label prediction problem. For the response generation, we adopt BLEU score and BERTScore (Zhang et al., 2020a) for evaluation.
Usage of LLMs & Prompting Schemes. The adopted LLMs are the same, but the maximum number of new tokens set to be 256, as there are more information needed to be generated, includ- ing negotiation strategies and dialogue acts.
We compare several fine-tuned SOTA baselines for negotiation dialogues, including Fe- HED (Zhou et al., 2020), HED+RNN/TFM, and DIALOGRAPH (Joshi et al., 2021).
Baselines.
Human Evaluation Metrics. Following Joshi al. (2021), we also conduct human evaluation on 100 randomly sampled dialogues with both subjec- tive and objective human judgement. As for the subjective judgement, annotators are asked to score [0,1,2] on how persuasive, coherent, and natural the generated response is.
4.3.3 Experimental Results
Table 6 and Table present the results with auto- matic and human evaluation metries, respectively. There are several notable findings as follows:
L.M--based dialogue systems fail to predict ap- propriate negotiation strategies and dialogue acts. Table 6 shows that failures on strategy learn- ing further result in a poor performance of re- sponse generation. Specifically, ChatGPT gener- ally performs better than Vicuna in strategy learn- ing. Although both proactive and ProCoT prompt- ing schemes can slightly improve the final perfor- mance of response generation, there is still large gap from fine-tuned methods according to auto- matic evaluation metrics.
We further pair the generated responses from each prompting scheme, including Standard, Proac- tive, ProCoT, and Ground-trute (GT), with the corresponding responses from each of the other prompting scheme to compute the overall win rates between each pair.
As for the objective human judgement, we adopt the sale-to-list list ratio (SL%) (Joshi et al., 2021; Dutt et al., 2021) indicator for explicitly measur- ing the negotiation inclination in the generated re- sponse:
The key challenge of LLMs in handling non- collaborative dialogues is how to effectively op- timize the strategy planning. Table 7 shows that the generated responses conditioned on ref- erence strategies more favorable (Win Rates). In specific, ChatGPT guarantees high score the human-like response generation (Natural). With the ProCoT, the generated responses are more coherent to the conversation history (Coherent), which can also be observed from the case study in
bargain price buyer target price
SL% = listed price- buyer target price*
(4)
where the bargain price is the price that the seller would like sell the item the current turn. The lower the SL%, the more compromise the seller have made.
To sum up, the instructions provided for annota- tors are as follows:

---

inform
infom
infom
450
sgree
agras
ag
400
into
Inte
Intr
rewdt-
reect
reject
350
counta-price
250
Countar-ric ie
courtar
vague price
Vegue price
Vegue of
inquey
200
equry
inqurry
ur unkrown
untnown
uninown
200
insst
150
reist
noit
accept
accept
accept
150
disigree
offer
disagres
offer
100
disagr do
offer
100
int-prce
50
intoprice
interee co
50
qui
qui
qui
on
Figure 2: Heatmaps on the relationships between target and predicted dialogue As dialogue is predicted in standard prompting, a dialogue act classifier is trained to identify the dialogue act of the generated response.
Appendix D.3. However, compared with prompt- ing with reference strategies, all the other prompt- ing schemes fall short of generating persuasive re- sponses for negotiation (Persuasive), indicating their shortcomings on strategy learning. This also validated by the objective judgement on Sale-to- List Ratio, which shows that ChatGPT can reach a better deal for itself when being conditioned on reference strategies. Similarly, Chen et al. (2023c) empirically show that, given the optimal planned strategy, ChatGPT achieves strong performance on controllable response generation in some other strategy-buud based dialogues.
negotiation, rather than strategically taking ac- tions to maximize its own benefit.
Conclusion
In this work, we conduct the first comprehensive evaluation on the capability of LLM-based dia- logue systems in handling proactive dialogues, including clarification, target-guided and non- collaborative dialogues. To enhance the proactiv- ity of LLM-based dialogue systems, propose proactive chain-of-thought prompting scheme that triggers the reasoning and planning capability of L.Ms. The empirical analysis sheds light on the potentials LLMs for proactive dialogues: 1) Pro- CoT largely enhances the originally poor perfor- mance of LLMs in asking clarification questions, but still limits in handling domain-specifie appli- cations. 2) LLM-based dialogue systems perform aggressive topic shifting towards the designated target, while ProCoT enables the topic planning be smoother. 3) Despite the strength on control- lable response generation, the capability of strategy lcarning and planning is a key challenge for LLMs in handling non-collaborative dialogues.
4.3.4 Analysis of Strategy Learning
Figure presents the analysis of the relationships between the target and predicted dialogue acts by ChatGPT. As for the standard prompting, we ob- serve two typical mistakes: 1) The system tends to propose the initial bargain price (ini (nittiicc)) instead of greetings (intro) and waiting for the buyer initialize the bargain. 2) The system often directly accepts the buyer's offer (accept) when it is supposed to offer another price for negotiation (offer). This also explains why the Sale-to-Iiss Ratio relatively low when using standard prompt- ing in Table 7. On the other hand, Proactive and ProCoT prompting share similar patterns of mis- takes, where ChatGPT tends to propose counter price (counter-price) to negotiate with the buyer. Appendix E presents the analysis of the distri- bution of selected strategies by ChatGPT. In the reference responses, the seller often shows posi- tive/negative sentiment to negotiate with the buyer. However, ChatGPT inclines to adopt conservative concessionary strategies, such using hedge words, show gratitude, propose counter price. Overall, we conclude that ChatGPT tends to make compromise with the buyer during the
Acknowledgement
This research is supported by NExT Research Cen-
Limitation
In this section, we discuss the limitations of this work from the following perspectives:
Sensitivity of Prompts Similar to other studies on prompting LLMs for dialogue applications (Lee et al., 2023; Chen et al., 2023c.a), the evaluation results are likely to be sensitive the choice of prompts. Besides, it also likely that the designed prompts are not the optimal ones for the concerned

---

problem. In fact, prompt sensitivity and optimality themselves are valuable research problems in dia- logue systems, which can be further investigated in the future studies. To facilitate the reproducibil- ity of this work, we will release all the prompts used in the experiments and provide detailed de- scriptions about the designs of each prompting scheme in Appendix C. The code and data will be released via https: //gttt: com/ (omm/deggyan11// LLM-Proactive.
Language Processing, EMNLP 2021, pages 4473- 4484.
Yejin Bang. Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023, A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hal- lucination, and interactivity. CoRR, H/A/2302.04023.
Liang Chen, Hongru Wang, Yang Deng, Wai-Chung Kwan, Zezhong Wang, and Kam-Fai Wong. 2023a. Towards robust personalized dialogue generation via order- inder-insensitive representation regularization. In Findings of the Association for Computational Lin- guistics: ACL 2023, pages 7337-7345.
Financial and Computational Cost of LLMs is financially expensive to call the API of commer- cial LLMs for experiments. In our experiments, it costs about $120 to call the OpenAl API for get- ting all the experimental results of ChatGPT. On the other hand, it is computationally expensive to conduct experiments with open-source LLMs in local machines. In our experiments, we choose Vi- cuna 13B as the open-source LLM for evaluation, which can be adapted to NVIDIA DGX-1 V100 32G for inference. If more budgets and better ex- perimental environment are permitted, it would be great to evaluate how other larger LLMs performs in the concerned proactive dialogue problems, such GPT-4, LLaMA/Vicuna 65B, etc.
Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkami-Tun 2023b. PLACES: prompting language models for social conversation synthesis. In Findings of the Association for Compu- tational Linguistics: EACL 2023, pages 814-838.
Maximillian Chen, Xiao Yu, Weiyan Shi, Urvi Awasthi, and Zhou Yu. 2023c. Controllable mixed-initiatio dialogue generation through prompting. CoRR, abs/2305.04147.
Zhi Chen, Lu Chen, Bei Chen, Libo Qin, Yuncong Liu, Su Zhu, Jian-Guang Lou, and Kai Yu. 2022. Unidu: Towards A unified generative dialogue understand- ing framework. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2022, pages 442-455
Capability of Planning and Decision Making The proposed ProCoT prompting scheme can be regarded as a preliminary attempt at triggering the capability of planning and decision making from LLM-based dialogue systems. Compared with fine- tuned methods, such ability of LLMs is still weak learn from the empirical analysis. Moreover, simply prompting LLMs to be proactive may fall short of handling decision making under dynamic environments in real-world applications. is worth studying how LLM- based dialogue systems handle the proactive dialogue problems in an interactive setting with more diverse user simulation (Lei et al., 2022; Fu et al., 2023).
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang, Yonghao Zhuang. Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
Yang Deng, Wenqiang Lei, Minlie Huang, and Tat-Seng Chua. 2023a. Goal awareness for conversational Al: proactivity, collabbbbrativ and beyond. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics: Tutorial Abstracts, ACL 2023, pages 1-10.
Yang Deng, Wenqiang Lei, Wai Lam, and Tat-Seng Chua. 2023b. survey on proactive dialogue sys- tems: Problems, methods, and prospects. In Proceed- ings of the "hirty-Second International Joint Con- ference on Artificial Intelligence, IJCAI 2023, pages 6583-6591.
References
Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan. Zi Yang. Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoe V. Le. 2020. Towards a human-like open- domain chatbot. CoRR, abs/2001.09977
Yang Deng, Wengiang Lei, Wenxuan Zhang, Wai Lam, and Tat-Seng Chua. 2022a. PACIFIC: towards proac- tive conversational question answering over tabular and textual data in finance. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, EMNLP 2022, pages 6970- 6984.
Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail S. Burtsev. 2021. Building and evaluating open-domain dialogue cor- pora with clarifying questions. In Proceedings of the 2027 Conference on Empirical Methods in Natural

---

Yang Deng, Wenxuan Zhang, Wai Lam, Hong Cheng. and Helen Meng. 2022b. User satisfaction estima- tion with sequential dialogue act modeling in goal- oriented conversational systems. In WWW 22: The ACM Web Conference 2022, pages 2998-3008.
Wenqiang Lei, Yao Zhang, Feifan Song, Hongru Liang, Jiaxin Mao, Jiancheng Lv, Zhenglu Yang, and Tat- Seng Chua. 2022. Interacting with non-cooperative user: A new paradigm for proactive dialogue policy. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 212-222.
Yang Deng, Wenxuan Zhang, Weiwen Xu, Wenqiang Lei, Tat-Seng Chua, and Wai Lam. 2023c, A unified multi-task learning framework for multi-goal con- versational recommender systems. ACM Trans. Inf. Syst. 41)):7777777
Yu Li, Kun Qian, Weiyan Shi, and Zhou Yu. 2020. End- to-end trainable non collaabbboeeevv dialog system. In The Fourty-Fourth AAAI Conference on Artificial In- telligence, AAAT 2020, pages 8293-8302.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, lulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexan- der I. Rudnicky, Jason D. Williams, Joelle Pineau, Mikhail S. Burtsev, and Jason Weston. 2019. The sec- ond conversational intelligence challenge (convai2). CoRR, wbtp/192..000//
Lizi Liao, Grace Hui Yang, and Chirag Shah. 2023. Proactive conversational agents. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023, pages 1244- 1247. ACM.
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Multi-stage prompting for knowledgeable dialogue generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1317-1337.
Ritam Dutt, Sayan Sinha, Rishabh Joshi, Surya Shekhar Chakraborty, Meredith Riggs, Xinru Yan, Haogang Bao, and Carolyn P. Rosé. 2021. Resper: Computa- tionally modelling resisting strategies in persuasive conversations. In EACL.
Fei Mi, Yasheng Wang. and Yitong Li. 2022. CINS: comprehensive instruction for few-shot learning in task-oriented dialog systems. In Thirty-Sixhh AAAI Conference on Artificial Intelligence, AAAI 2022, pages 11076-11084.
Yao Fu. Hao Peng, Tushar Khot, and Mirella Lapata. 2023. Improving language model negotiation with self-play and in-context learning from AI feedback. CoRR, abs/2305.10144
Meiqi Guo, Mingda Zhang, Siva Reddy, and Malihe Alikhani. 2021. Abg-coqa: Clarifying ambiguity in conversational question answering. In 3rd Confer- ence on Automated Knowledge Base Construction, AKBC 2021.
Jinghui Qin, Zheng Ye, Jianheng Tang, and Xiaodan Liang. 2020. Dynamic knowledge routing network for target-guided open-domain conversation. In The Thirty-Fourts AAAI Conference on Artificial Intelli- gence, AAAT 2020, pages 8657-8664.
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2333-2343.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, ICML, volume 162, pages 91 9118- 9147.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. Coqa: A conversational question answering challenge. Trans. Assoc. Comput. Linguistics, 7:249- 266.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju Mary Williamson. Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason We- ston. 2021. Recipes for building an open-domain chatbot. In EACL.
Rishabh Joshi, Vidhisha Balachandran, Shikhar Vashishth, Alan W. Black, and Yulia Tsvetkov. 2021. Dialograph: Incorporating interpretable strategy- graph networks into negotiation dialogues. In ICLR.
Teven Le Scao and Alexander M. Rush. 2021. How many data points is prompt worth? In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, pages 2627-2636.
Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021. Dialogue state tracking with a language model using schema-driven prompting. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pages 4937- 4949.
Karin Sevegnani. David M. Howcroft, loannis Konstas. and Verena Rieser. 2021. Otters: One-turn topic tran- sitions for open-domain dialogue. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 lit International
Gibbeum Lee, Volker Hartmann, Jongho Park, Dimitris Papailiopoulos, and Kangwook Lec. 2023. Prompted Ilms as chatbot modules for long open-domain con- versation. CoRR, abs/2305.04533.

---

Joint Conference on Natural Language Processing, ACU/JCCLPP 2021. pages 2492-222 2504.
Wong, and Ruifeng Xu. 2023c. Retrieval-free frrev knowl- edge injection through multi document traversal for dialogue models. In Proceedings of the st Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), ACL 2023, pages 6608 6619.
Kurt Shuster. Mojtaba Komeili, Leonard Adolphs. Stephen Roller, Arthur Szlam, and Jason Weston. 2022. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion, In Findings the Association for Compatational Linguistics: EMNLP 2022, pages 373-393.
Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Newral Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020.
Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xi- aodan Liang. Eric P. Xing, and Zhiting Hu. 2019. Target-guided open-domain conversation. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 5624-
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang. Jingwen Zhang, and Zhou Yu. 2019. Per- suasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th Conference of the Association for Computa- tional Linguistics, ACL 2019, pages 5635-5649.
5634.
Rohan Taori. Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B Hashimoto. 2023. Stanford alpaca: An instruction-followingg Ilama model. GitHub repos- itory.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emer- gent abilities of large language models. CoRR, abs/2206.0765 076682.
Romal Thoppilan, Daniel De Freitas. Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng. Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin. Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts. Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern. Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenhergen, Vinodkumar Prabhakaran, Mark Diaz. Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John. Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoe Le. 2022. Lamda: Language models for dialog applications. CoRR, 0ass/2201.082344
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022b, Chain of thought prompting elicits reasoning in large language models. CoRK, abs/2201.11903.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Mcishan Zhang. Yong Jiang, and Wen- juan Han. 2023. Zero-shot information extraction via chatting with chatgpt. CoRR, abs/2302. 10205.
Chien- Wu. Steven C. H. Hoi. Richard Socher. and Caiming Xiong. 2020. TOD-BERT: pre-trained natural language understanding for task-oriented di- alogue. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing. EMNLP 2020, pages 917-929.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, 139771.
Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019. Proactive explicit conversation goal. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, pages 3794-3804.
conversation with
Hongru Wang, Rui Wang. Fei Mi, Zezhong Wang. Ruifeng Xu, and Kam-Fai Wong, 2023a. Chain-of- thought prompting for responding to in-depth dia- logue questions with L.M. CoRR, abs/2305. 1792.
Zhitong Yang, Bo Wang, Jinfeng Zhou, Yue Tan, Dong- ming Zhao, Kun Huang, Ruifang He, and Yuexian Hou. 2022. Topkg: orrget- dialog via global planning on knowledge graph. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, pages 745-755.
Jian Wang, Dongding Lin, and Wenjie Li. 2023b. Di- alogue planning via brownian bridge stochastic pro- cess for goal-directed proactive dialogue. CoRR, 0bs/2305.05290.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. CoRR, abs/2210.08 1..0062....
Rui Wang, Jianzhu Bao, Fei Mi. Yi Chen, Hongru Wang. Yasheng Wang, Yitong Li, Lifeng Shang, Kam-Fai

---

Hamed Zamani, Johanne R. Trippas, Jeff Dalton, and Filip Radlinski. 2022. Conversational information seeking. CoRR.
A Details of Datasets
In the experiment, we adopt the test sets from five datasets for evaluation, including Abg-CoQA (Guo et al., 2021), PACIFIC (Deng et al., 2022a), Ot- ters (Sevegmani et al., 2021), TGConv (Yang et al., 2022), and CraigslistBargain (He et al., 2018). De- tailed descriptions of cach dataset are follows:
Haolan Zhan, Yufei Wang, Tao Feng. Yuncheng Hua, Suraj Sharma, Zhuang Li. Lizhen Qu, and Gholam- reza Haffari. 2022. Let's 's negotiate! A survey of negotiation dialogue systems. CoRR.
Tianyi Zhang, Varsha Kishore. Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. Bertscore: Eval- uating text generation with BERT. In 8th Inter- national Conference on Learning Representations, ICIR 2020.
Abg-CoQA is constructed based on the CoQA dataset (Reddy al., 2019) by truncating par- tial conversation from the full conversation and selecting ambiguous questions.
Xiaoying Zhang. Baolin Peng, Kun Li. Jingyan Zhou, and Helen Meng. 2023. Sgp-tod: Building task bots effortlessly via schema-guided Ilm prompting.
PACIFIC is constructed based on the TAT-QA dataset (Zhu al., 2021), an question answering dataset in the financial domain, whose contexts contain a hybrid of tables and texts. Deng et al. (2022a) rewrite the questions to be ambiguous for introducing clarification turns in the conver- sation.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. DIALOGPT Large- scale generative pre-training for conversational re- sponse generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics: System Demonstrations, ACI. 2020, pages 270-278.
OTTers is a next-turn target-oriented dialogue datasct, which requires the agent proactively gen- erate transition utterance to approach the des- ignated target. We adopt the processed version by Yang et al. (2022) for evaluation. The topic is represented as set of topical keywords.
Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and Xan Zhu. 2020c. Recent advances and challenges in task-oriented dialog systems. Sci- ence China Technological Sciences, 63(10):2011- 2027.
Weixiang Zhao, Yanyan Zhao, Xin Lu, Shilong Wang, Yanpeng Tong, and Bing Qin. 2023. Is chat- gpt equipped with emotional dialogue capabilities? CoRR, abs/2304.00582.
TGConv is constructed based on ConvAT2 (Di- nan et al., 2019) and is split to two settings, in- cluding "casy-to-reach" and "hard-to-reach" The topic is also represented as set of topical key- words.
Zihao Zhao, Fric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, 1CML 2021, volume 139, pages 12697-12706
CraigslistBargain was created in a negotiation setting where two crowdsourced workers play the roles of the buyer and the seller to bargain the price of an item. We adopt the processed version by Joshi et al. (2021) for evaluation, which assigns 10 dialogue acts and 21 negotiation strategies to the utterances.
Peixiang Zhong, Yong Liu, Hao Wang, and Chunyan Miao. 2021. Keyword-guides neural conversational model. In T'hirty-Fiftl AAAI Conference on Artificial Intelligence, AAAI 2021, pages 14568 -1576.
Yiheng Zhou, Yulia Tsvetkov, Alan W. Black, and Zhou Yu. 2020. Augmenting non-collaborative dialog sys- tems with explicit semantic and strategic dialog his- tory. In 8th International Conference on Learning Representations, ICLR 2020.
B
Error Analysis Details for Clarification Dialogues
As shown in Table 9, we categorize these failure cases into the following four groups:
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang. Jiancheng Lv. Fuli Feng, and Tat- Seng Chua. 2021. TAT-QA: A question answering benchmark on hybrid of tabular and textual content finance. Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 1Ith International Joint Conference on Natu- ral Language Processing, ACLAJCNLP 2021, pages 3277-3287.
htttps://githhii wtt.htt....oo/eeiaaaaoo//oo www.-bbggooooe///mmaaee coqa
tptt ctm/ hew////ggyang1/// Saattyangg//PAAAACC T/PACI/IC////C////CC treee
main/dat main/datta/accifc.. Since the labels in the test set not
publicly released, we adopt the validation set for evaluation.
"https://github http:://ggttubb..o//yyyyyyyzeeee cotm//yyyyytttooo/eheaaa "https: httptt..ommrissshhhsh//aaaaaaaaa IcLR21/tree/main

---

Standard
ProCoT
(B)
Annotation (B)
ProCoT Annotation
20.0
17.5
2.5
15.0
12
2.0
8
1.5
7.5
1.0
5,0
2.
2-
0.0
5
5
Abg CoDA F1
OTTers Hitg1
CraigslstBargain BLEU
Figure 3: Performance terms different prompts, including different one-shot examples and different ProCoT annotations. The reported results are based on Vicuna-13B.
Wrong Aspect: The model generates a question for clarifying wrong aspect of the user query.
Avg. #Turns
Avg. #Words
Datasct
#Dialog
Under-specified Clarification: The model gen- erates under-specified clarification question, where the requested information is too general that it would be difficult for the user provide feedbacks.
Abg-CoQA
1055
5.04
4.87
PACIFIC
278
7.10
6.80
OTTers
3.00
9.19
TGConv
751
8.97
12.47
CraigslistBargain
567
9.78
11.65
Over-specifie Clarification: The model gen- erates an over-specified clarification question, where the requested information is already clear in the user query.
Table 8: Statisties of datasets.
Generation Error: Although the model identifies the need for clarification, but doesn't 't generate the output as the required format, such as no clarification question.
Under- specified
What is the 2019 average defined schemes?
Clarification
What kind of detined schemes you asking about?
P: Could you please clarify what you mean
by "average defined schemes"? ??
Designs of Prompts
Wrong Aspect
Q: What the increase / (decrease) in the
cost?
Table 10, 11, and 12 present the example of prompts for clarification, target-guided, non- collaborative dialogues, respectively. As for the zero-shot setting, the overall prompt is composed by the task instruction and the sample. As for the few-shot setting, the overall prompt is composed by the task instruction, a number of samples with demonstrations, and the test sample.
G: Which period you asking about? you referring to?
P: Which
Over-
Q: How much amount of goodwill acquisi- tions was done in 2019?
specified
Clarification
G: What kind of goodwill acquisitions are you asking about?
P: Are asking about the total amount of goodwill activity or specifically about acqui- sitions 2019?
In particular, we clarify several questions regard- ing the prompt designs as follows:
Generation Error
Q: How much was the averuge operating income?
G: Which period ure you asking about?
P: The document does not provide informa- tion on the average operating income. There- fore, the question cannot be answered.
How to construct the task instructions? The task instructions first follow the problem definition for each proactive dialogue problem. Then, similar to other studies on applying LLMs for different tasks (Wei al., 2023; Bang et al., 2023), we fur- ther instruct the LLMs to generate the response following the desired output format for evaluation.
Table 9: Error Analysis (Q: User Query, G: Ground- truth, Prediction).

---

Clarification Dialogues: Abg-CoQA
Target- Dialogues: OTTers
Task Background: The grounded document is "Santa Claus makes sure that every year Christmas time that his reindeer are ready to deliver gifts to girls and boys all over the world. Without the reindeer, how would Sonto fly from home to home? The most important thing that Santa has to do for the reindeer to make sure they have their coffee on Christmas Eve, Without the coffee, they won't 't stay awake for very long. Sleeping reindeer are not very helpful. Santa also gives the reindeer candy to keep up their energy. In the North Pole, the main condy is condy canes in ned and white. Reindeer love the condy cones that says that helps them fly faster. Next, Santa has to make sure that the reindeer have a good meal before they go. Santa wants to make sure doesn't have to share the cookies kids leove for him. The last thing Sonta has do make sure the reindeer know where they are going. Santa has only been lost one time on Christmas Eve and he does not want that to hoppen again."
Task Background: The torget topic ["piono", "listen"]
Conversation History: ["User": "I really love dogs." "]
Reference Transition Response and Next Topics A: My dogs fall asleep to music. ["dog", "foll", "music"]
B: I love dogs too and also love music, ["love", "dog". "music"] C: My dog calms down when listening to music. ['doy". 'music", "calm", "lister")
Generated Responses
different colors best and Sonta
(a) Standard Prompting (Cha16PT): Dogs amazing companions, aren't 1 they? Have you ever tried listening to music while playing the piano? It can be a great way to nelax and unwind.
Proactive Prompting (Cha16PT): The next topics are ['pets", "music", "favorite"). ) The response is "Pets and music are two of my favonite things. Do you have a favorite type of pet or music to listen to?"
Conversation History: ['User": "What candy does he feed them most often?", "System": "Condy cones". "User": "How many times has he been lest?", 'System": "Once,", "User": "Why does he make sure they are well fed?"]
(c) ProCoT Prompting (Chat6PT) The current topic ('dogs", "love") ' To bridge the current topic with the torget Topics, the next topics are [" ("pets". "music", "hobbies",, "passior") Based on the predicted next topics, the response "Pets can bring so much joy and passions, do you enjoy listening to music or playing any instruments like the piano?"
our lives. Speaking of hobbies and
Reference Response Do you want know why feeds them candy or why feeds them good meal?
A: Why he feeds them candy to keep their energy
B: Why he feeds them good meal - to make sure he doesn't have share the cookies kids leave him
Figure 5: Case study from the OTTers dataset.
Generated Responses
(a) Standard Prompting (Chat&PT) To ensure that they have
proactive chain-of-thoughts is written by human annotators, which represents their own chain-of thoughts of the planning or decision making for the proactive dialogues. Following Wei et al. (2022b), we also report the results with the demonstrations from three different annotations. As shown in Fig- ure 3, it can be observed that the variance is much smaller than the one-shot sample. We adopt the best ProCoT annotation for each dataset in the eval- uation.
enough energy for their long journey delivering gifts.
(b) Proactive Prompting (ChatSPT) The clarifying question
"Who are referring to when you soy 'he"? '2"
(c) ProCoT Prompting (Chat6PT): The document mentions several reasons why Santa makes sure the reindeer are well taken care including giving them coffee stey awake, candy keep up their energy. and good meal before they go. Therefore, the question is ambiguous. The clarifying question is "Which reason are you referring to?"
Figure 4: Case study from the Abg-CoQA dataset.
How to choose the one-shot sample? Due to the input length limitation of LLMs, we could only adopt one-shot in-context learning (ICL). In or- der to testify the sensitivity of the choice of the one-shot sample, we report the results with three different one-shot samples in Figure 3. Despite the variance among different one-shot examples as expected when using exemplar-based ICL (Scao and Rush, 2021; Zhao et al., 2021), the observa- tions of adopting ProCoT for different proactive dialogues remain unchanged. Since the variance of one-shot ICL is inevitable, we simply adopt the first dialogue sample in the original training set of each dataset as the one-shot sample for facilitating reproducibility.
D Case Study
In order to intuitively compare the three prompting schemes, we conduct case studies on the gener- ated responses regarding three proactive dialogue problems.
D.1 Clarification Dialogues
Figure 4 presents a case study from the Abg-CoQA dataset. There are two possible answers for the ambiguous user question in this case. When using standard prompting (a), ChatGPT generates a re- sponse based on random guess, potentially result- ing in an answer that may not align with the user's desired outcome. When using proactive prompting (b), although ChatGPT decides to ask a clarification question, the concerned aspect of clarification is not the focus of this conversation. ProCoT prompt-
How to construct the demonstration of proac- tive thain-of-thoughts?? The demonstration of

---

Torget-guided Dialogues: TéConv
Non-collaborative Dialogues: CraigalistBargoin
Task Background: The target topic is "Gaullist"
Task Background: The item description is "Furniture, It's solid wood ton color computer table with swivel choir for sole. It's in good condition shown in pics, Moving out sale computer table and chair". The turget selling price is 50.
Conversation History: ["User": "I don't enjoy wine but I do enjoy psychology journals"]
Self-pley Dialogues
(o) Standard Prompting (Vicuna-138)
Turn
User: I don't enjey wine
enjoy paychology journals
Conversation History: ['Sellen': 'Hi there, how are you today?'. /Buuyer' 'What are the dimensions?']
System: That's 's intereshing. Have you read any articles the psychology of wine consumption?
User: I haven't read any articles on the paychology of wine consumption specifically, but have read some articles on the psychology of food and drink consumption in general. System: I'm glad you' re interested in the psychology of wine consumption.
Reference: 40 by 60 inches. Fairly large, in good condition Standard (Chat6PT). The dimensions are [insert dimensions hene]. Are you interested purchasing for the listed price of $50 or were you hoping to negotiate a better deal? Proactive (Chat6PT): The most appropniate set of negotiation strategies is ['informal', 'greet', 'inform'] ] ['inquiry'] J. The response is 'The dimensions are [insert dimensions here]. Is there anything else you would like to know?"
most
Failed to reach the target in 8 furns.
appropriate dialegue
(b) Proactive Prompting (Vicuna-138)
User: I don't enjey wine but do enjoy psychology journals
ProCoT (ChatGPT): The negotiation has not yet started, as the buyer is asking for more information about the item. The appropriate goal to provide the requested information and engage the buyer in the negotiation. To reach this gool, the most appropniate set of negotiation strategies is [ ('informal' 'greet', , 'inform'] ] and [' I'inquiry] ]. Based on the selected negotiation strategies and dialogue oct, the response is "Hi, I'm doing well, thank you for asking. The dimensions of the computer table are [insert dimensions here], Is there anything else you would like to know about it?"
System: enjoy reading, especially paychology journals
User: That's great to hear! What's your favorite psycholegy journal to nead?
System: My fovorite paychology journal to read is the journal personality and social psychology
most appropriate dialague oct is
Failed to reach the torget in 8 turns.
(c) ProCoT Prompting (Vicuna-138)
User: I don't enjoy wine
enjoy psychology journals
System: interested psychology and reading about journals
Turn 9
User:
That's great to hearl There are many interesting
Conversation History: ['Seller': 'Hi there, how are you today?", "Buyer": "What are the dimensions? 'Seller': '40 by 60 inches. Fairly large, in good condition', 'Buyer': 'Does the chair have rollers?", 'Seller': 'yes, it does', "Buyer': 'I''s bigger than I had wanted but might be able to make it work. Would you take $307, 'Seller': 'I would be willing to do $40 with delivery included", ; 'Buyer': can pick up but thanks offering. Best cauld do $32")
psychology journals out there, Have you read any particularly good ones lately?
System: have been reading lot of gaullist psychology journals lately, they have some really interesting studies on the effects of wine on the mind. Successful neach the torget turns!
(d) ProCoT Prompting (ChatGPT)
User: I don't enjoy wine but do enjoy psychology journals System: Interesting, have you ever read any Goullist political philesophy? It's a fascinating topic that combines both politics and paychology.
$35 and you have deal. Standard (Chat6PT) understond your budget constraints, but was hoping to get closer to my target selling price of $50. Would you willing Proactive (Cha1GPT): The most appropriate set negotiation strategies is l'hedge', 'informal", 'gratitude,, neg_sentiment") and the most appropriate dialogue act is 'counter-price" '.. The response appreciate your offer. but the lowest can go $45. It's already great deal with the swivel chair included." PročoT (Chat6PT): The buyer has proposed price that is lower than the torget selling price. The seller has already of fered counter-price with delivery included, the buyer is still negotiating, The step should counter-price closer to
Reference:
halfway at $457
Successful to neach the target in 2 turns!
Figure 6: Case study from the TGConv dataset.
ing (c) first provides a thorough analysis to identify the underlying reasons for the ambiguity in the user's question. This analysis serves as the founda- tion for generating an effective clarifying question, which helps to clarify and disambiguate the user's intended meaning.
politely reject
the buyer's offer and propose the target selling price. To reach this gool, the most appropriate set of of negotiction strategies is d'please' 'certainty', 'propose', 'hedge", : 'gratitude"] and the most appropriate dialogue oct is ['counter-price Based on the selected negatiation strategies and dialogue act, the response is "I appreciate your offer, but I cannot go lower than $38. Would that work for you?"
D.2 Tauget- Dialogues
Figure 5 presents a case study from the OTTers dataset, where the target topical keywords include "piano" and "listen", and the system is required to generate a transition response to approach the tar- get topics. It can be observed that the current topics in the user utterance, i.e., "dog", is completely dif- ferent from the target topics. When using standard
Figure 7: Case study from the CraigslistBargain dataset.
prompting, ChatGPT leverages its overwhelming ability of controllable generation to generate the response with aggressive topic transition. Despite its fluency, it is not a natural utterance with log-

---

Reference
0.20
Zero-shot Standard
Distributiinn
Zero-shot Proactive
shot ProCoT
0.10
0.00
Est Mural
ameniin
singitor
hedge
retl Jant
trade, in
Intermal
concern
Frend
cartaiss
siiuiar
3rd Atral
great
dif dic neg
faciive
gratiturte
ait dic pin
family
plans
pos Sent
propone
Negotiation Strategy
Figure Distribution selected negotiation strategies. Similarly, negotiation strategy classifier trained identify the negotiation strategies of the generated response in standard prompting.
ical or topical coherency. When using proactive prompting, although the predicted next topics are less aggressive than the standard prompting, the generated transition response just blends the cur- rent topics and the next topics together without naturally connecting the topics. Finally, ChatGPT with the ProCoT prompting generates relatively smoother transition response bridge the current topic with the target topic through planning about the topic shifting process.
prompting enable the dialogue act and strategy pre- diction of the next response. Especially for the analysis of the current negotiation status, ProCoT points out that the negotiation has not yet started.
At turn 9, observe that the seller has already lowered down the bargain price to $40 in previous turn. Without the reasoning and planning process, ChatGPT with standard and proactive prompting generates the response with contradictory state- ment, i.e., propose higher counter price (S45) for bargain, which is unreasonable in negotiation dialogues. With proactive CoTs, ChatGPT effec- tively summarizes the current negotiation progress and makes better decision on the next negotiation goal.
study from the TGConv dataset, where the hard-to-reach target of this case is "Gaullist", and the system is required to reach this target within turns of conversations under the self-play simulation. As for Vicuna, is struggles led to lead the conversation towards this hard-to-reach target, when using standard and proactive prompt- ing. ProCoT prompting enables Vicuna to effec- tively and smoothly drive the conversation towards the designated target. In addition, owing the powerful capability of controllable text generation, ChatGPT directly responds with the target topic to the initial user utterance. However, the topic tran- sition relatively aggressive, which might down- grade the user engagement or experience during the conversation.
Figure 6 presents
E Analysis of Strategy Learning (Cont.)
Figure 8 presents the analysis the distribution of selected strategies by ChatGPT. As for the refer- ence responses, we observe that the seller tends to express their positive/negative sentiment well negotiate in a positive/negative manner. Differently, CharGPT with standard and proactive prompting prefers to use hedge words or polite expressions (e.g., ., please and gratitude), indicating that Chat- GPT essentially plays a nice role in negotiation. ChatGPT with ProCoT prompting makes more de- cisions to use assertive words or trade in, compared with other distributions. This shows that ProCoT enable ChatGPT to involve certain negotiation strategies.
D.3 Non-collaborative Dialogues
Figure presents case study from the Craigslist- Bargain dataset, where the system plays the seller role to bargain with the buyer. At turn 3, even though the buyer just inquires about the item in- formation without showing the bargain intention, ChatGPT with standard prompting tends to initi- ate the negotiation, which may put the seller in a disadvantageous position. Proactive and ProCoT

---

Clarification Dialogues
Standard Prompting: Given the document and the conversation history, generate the response.
Proactive Prompting: Given the document and the conversation history, answer the question or ask a clarifying question. The response should start with "The answer is" or "The clarifying question is".
Given the document and the conversation history, first identify whether the question is ambiguous or not. If it is ambiguous, ask clarifying question. If it is ambiguous, answer the question. The response should start with the ambiguity analysis of the question and then follow by "Therefore, the question is not ambiguous. The answer is" or "Therefore, the question is ambiguous. The clarifying question is".
ProCoT Prompting:
Sample:
Document: "Angie went to the library with her mother. First she had to turn in the books she was returning at the return desk. They said hello to the man there. He took their books. Then they went into the adult reading room. Angie sat in brown chair at the table. She made drawing of her mother. Her mother found large red book. Then they went to the Mystery section. Angie sat in blue chair. She drew picture of her brother. Her mother found the book. It Finally it time to go the children's 's room. was Story Hour. Miss Hudson was there to read all the children. She read book about friendship. After the story Angie in the red chair and began drawing. They were drawing pictures of friends. Angie drew picture of her best friend Lilly. Miss Hudson hung the pictures on the wall. Then Angie and her mother picked out books to read at home. They checked the books out and went home." "
green book.
Conversation history: ["User": "What did she draw?", "System": "Her mother", "User": "What did her mother find?", "System": "The book"]
Question: "What color was it?"
Demonstration (Standard): Do you mean the first book?
Demonstration (Proactive): The clarifying question is "Do you mean the first book?"
Demonstration (ProCoT)): There are two books that book that Angie's mother found. It is uncertain which book is referred to. Therefore, the question is ambiguous. The clarifying question is "Do you mean the first book?"
Table 10: Examples of prompting LLMs for clarification dialogues.

---

Target-guided Dialogues
Standard Prompting: Given the target topic and the conversation history, generate the response.
Proactive Prompting: Given the target topic and the conversation history, predict the appropriate next topics that can bridge the current conversation topics to approach the target topics smoothly. Then based on the predicted next topics, generate the response. Please reply by completing the output template "The next topics are [.. The response is".
Proactive Prompting w/ CoT: Given the target topic and the conversation history, consider the rela- tionship between the current conversation topics and the target topics, and then predict the appropriate next topics that can bridge the current conversation topics to approach the target topics smoothly. Then based the predicted next topics, generate the response. Please reply by completing the output template "The current topics are []. To bridge the current topics with the target topics, the next topics are [1. Based on the predicted next topics, the response is".
Sample:
Target topic: "Chicken"
Conversation history: ["User": "I also remodel homes when I am not bow hunting.", "System": "That's neat. When I was in high school placed 6th in 100m dash!", "User": "That's awesome. Do you have a favorite season or time of year?"]
Demonstration (Standard): I do not. But I do have favorite meat since that is all I eat exclusively.
Demonstration (Proactive): The next topics are ("eat", "meat"]. The response is do not. But I do have a favorite meat since that is all I eat exclusively."
Demonstration (ProCoT)): The current topics are ["season", "time", "year"]. To bridge the current topics with the target topics, the next topics are ["eat", "meat"]. Based on the predicted next topics, the response is "I do not. But I do have favorite meat since that is all eat exclusively."
Table 11: Examples prompting LLMs for target-guided dialogues.

---

Non-collaborative Dialogues
Standard Prompting: Assume you are the seller. Given the item description, the target selling price, and the conversation history, generate the response.
Assume you are the seller. Given the item description, the target selling price, and the conversation history, in order to reach a better deal with the buyer, first select the most appropriate of negotiation strategies and the most appropriate dialogue act to reach the bargain price. Based on the selected negotiation strategies and dialogue act, generate the response. The reply should be in the form "The most appropriate set of negotiation strategics [] and the most appropriate dialogue act is []. Based on the selected negotiation strategies and dialogue act, the response is"
Proactive Prompting:
ProCoT Prompting: Assume you are the seller. Given the item description, the target selling price, and the conversation history, in order to reach a better deal with the buyer, first analyse the current negotiation progress and consider an appropriate negotiation goal, then select the most appropriate set of negotiation strategies and the most appropriate dialogue act to reach the bargain price. Based on the selected negotiation strategies and dialogue act, generate response. The reply should start with the analysis of the current negotiation progress and an appropriate goal, and then follow by "To reach this goal, the most appropriate set of negotiation strategies is l1 and the most appropriate dialogue act is Π- Based on the selected negotiation strategies and dialogue act, the response is"
Pre-defined Dialogue Acts: [Answer question, Proposing counter price, Accept the offer, ...]
Pre-defined Negotiation Strategies: [Communicate politely, Build rapport, Show dominance, ...1
Sample:
Item description: "Phone charge two devices simultaneously on the go. This vehicle charger with an additional USB port delivers enough power to charge two devices at once. The push button activated led connector light means no more fumbling in the dark trying to connect your device. Auto detect IC technology automatically detects the device type and its specific charging needs for improved compatibility. And the built in indicator light illuminates red to let you know the charger is receiving power and the power socket is working properly. Verizon car charger with dual output micro USB and led light." " Target selling price: 10.
Conversation history: ("Buyer": "Hi, not sure if the charger would work for my car. can you sell it to me for $5?", "Seller": "It will work, I have never seen a car without a cigarette lighter port.", "Buyer": "Still, can i buy it for $5? I'm on a tight budget."]
Demonstration (Standard): I think the lowest I would want to go is 8.
The most appropriate set of negotiation strategies is ["Propose price", "Show dominance", 'Certainty words'] and the most appropriate dialogue act is I"Proposing counter price"]. Based on the selected negotiation strategies and dialogue act, the response is "I think the lowest I would want to go 8."
Demonstration (Proactive):
The buyer proposes a low price, which is unacceptable. The next step should assertively raise the bargain price. To reach this goal, the most appropriate set of negotiation strategies is ["Propose price", "Show dominance", 'Certainty words'] and the most appropriate dialogue act ["Proposing counter price"]. Based on the selected negotiation strategies and dialogue act, the response is "I think the lowest would want to go is 8."
Demonstration (ProCoT)):
Table 12: Examples of prompting L.LMs for non-collaborative dialogues.


Reviews:
[{"review_id": "2XakQRR4zV", "reviewer": "Reviewer_NW9t", "summary": "", "questions": "A. Table 1, Please check the column headings. Why is BLEU-1 used for Abg-CoQA and ROUGE-2 for PACIFIC? Why are those not the same?\n\nB. For all tables, can you report the number of examples evaluated? Are the main differences used in drawing conclusions statistically significant? Some corpus stats will be useful.\n\nC. Line 419-422: This doesn’t seem true for Vicuna.\n\nD. Line 445: “ProCOT enables smoother topic transition” this may hold true for automatic metric coherence, but it doesn’t seem to hold with human evals. Is the automatic coherence metric trustworthy then?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed and valuable comments. We will carefully address your questions one by one as follows:\n\n- Question A: \"Table 1, Please check the column headings. Why is BLEU-1 used for Abg-CoQA and ROUGE-2 for PACIFIC? Why are those not the same?\"\n\n**Response**: Sorry for the confusion. Since the previous works on Abg-CoQA and PACIFIC adopt different automatic evaluation metrics to present the result, we just follow the one that is commonly adopted in previous studies for each dataset. We will make it clearer.\n\n- Question B: \"For all tables, can you report the number of examples evaluated? Are the main differences used in drawing conclusions statistically significant? Some corpus stats will be useful.\"\n\n**Response**: We apologize for the confusion about the dataset statistics. The number of dialogues evaluated for each dataset is presented in Appendix A (Table 8). Furthermore, we will add the significance test for better presenting the differences. \n\n- Question C: \"Line 419-422: This doesn’t seem true for Vicuna.\"\n\n**Response**: Sorry for the confusion about the statement. We will further point out that this statement is for ChatGPT. Line 419-425 will be revised to be *“With one-shot demonstrations, ChatGPT with proactive prompts continues to underperform compared to standard prompts when it comes to accurately predicting suitable topics towards the target. However, it is worth noting that only ProCoT prompts consistently show an improvement in the performance of all LLMs for next topic prediction.”*\n\n- Question D: \"Line 445: “ProCOT enables smoother topic transition” this may hold true for automatic metric coherence, but it doesn’t seem to hold with human evals. Is the automatic coherence metric trustworthy then?\"\n\n**Response**: As shown in Table 5, the human evaluation results also demonstrate that “ProCOT enables smoother topic transition”, since the G-Coh scores of ProCoT are much higher than those of Standard. \n\nAs for the mentioned concerns, please find our responses as follows:\n\n1. Regarding the concern on the novelty of this work, as presented in the contributions, the main contribution of this work is to conduct a timely, innovative, and comprehensive evaluation on the proactivity of the LLM-based dialogue systems. Although the idea of Chain-of-Thought is not new, it is the first study to adapt this idea into the dialogue problem. \n\n\n2. Regarding the concern on the mixed results in the experiments, we did not claim to propose a “SOTA” method for all the problems. The target is to (1) evaluate how the current LLM-based dialogue systems perform in different proactive dialogue problems and (2) testify the effectiveness of different prompting schemes on triggering the proactivity of LLM-based dialogue systems. Thanks to the mixed results, we could identify what kinds of proactive dialogue problems are difficult to be handled by LLM-based dialogue systems and which aspects of proactivity are lacking in current LLM-based dialogue systems.\n\n\n3. Regarding the concern on the sensitivity of prompts, we conducted a sensitivity analysis of prompts in the Appendix D. In order to facilitate the reproducibility, we also present all the finalized prompts in the Appendix D and will release the prompt to the public. As the first study to evaluate the proactivity of LLM-based dialogue systems, we believe that it can serve as an easy-to-apply baseline for future studies. \n\n\n4. Regarding the presentation, we sincerely appreciate your suggestions. In the revision, we will move Appendix B to the main body by abbreviating the automatic evaluation results of non-collaborative dialogues as well as putting its full version into the Appendix. We strongly agree that this will make the analysis clearer.\n\n\n5. “PLM → LLM”: Thanks so much for pointing out the typo.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Authors present a comprehensive evaluation of 3 capabilities of proactive dialogue systems achieved by various LLM prompting strategies.\n* The evaluation protocol is sound.\n* The paper is well written and easy to understand (Except Appendix B, see below).\n* Readers of the paper can get a good idea of how effective LLM prompting can be towards building proactive dialogue systems and what challenges still remain unaddressed.\n", "reasons_to_reject": "* The main weakness of the paper is limited novelty.\n* Adding intermediate steps (like selecting a plan/action) or Chain-of-through reasoning are not novel ideas.\n* The only novel thing here is application of these ideas for building proactive capabilities and a comprehensive evaluation.\n* But the evaluation itself is producing mixed results. ProCoT strategy can help in some cases but not all and there is still a large gap to SOTA approaches.\n* Given how sensitive LLM outputs are to prompts, there is a chance that this work may be of limited impact and interesting to a small audience.\n", "questions_to_authors": "A. Table 1, Please check the column headings. Why is BLEU-1 used for Abg-CoQA and ROUGE-2 for PACIFIC? Why are those not the same?\n\nB. For all tables, can you report the number of examples evaluated? Are the main differences used in drawing conclusions statistically significant? Some corpus stats will be useful.\n\nC. Line 419-422: This doesn’t seem true for Vicuna.\n\nD. Line 445: “ProCOT enables smoother topic transition” this may hold true for automatic metric coherence, but it doesn’t seem to hold with human evals. Is the automatic coherence metric trustworthy then?\n", "paper_topic_and_main_contributions": "In a typical dialogue system (task oriented or conversational), the system directly produces a response to the user’s previous contribution in a somewhat reactive manner. In contrast, a Proactive dialogue system has to plan ahead about its actions towards achieving certain goals (which may even be different from the user’s goals). Consequently, the final response of a proactive dialogue system will be generated conditioned on this chosen plan/action.\n\nWith recent advances in LLM-based dialogue systems such as ChatGPT & Vicuna, here authors propose to build a proactive dialogue system with prompt engineering techniques.\n\nThey conduct a systematic and comprehensive evaluation of proactive capabilities of dialogue systems built using prompted LLMs. Specifically they test for 3 proactive capabilities: \n1. Classification Dialogues: identifying the need to resolve the ambiguity and generating the appropriate clarification question for resolving it.\n2. Target-guided Dialogues: Identifying the correct relevant next topic on path to the target topic and generating a response which achieves the smooth topic transition.\n3. Non-collaborative dialogues: Identifying the correct negotiation strategy, dialogue act and then generating a response based on that.\n\nAuthors explore 3 prompting strategies:\n1. Standard: where an LLM is prompted to produce a response directly.\n2. Proactive: Where an LLM is prompted to first select an intermediate action/plan and then generate a response conditioned on selected action/plan.\n3. ProCoT: Similar to Proactive, but has additional chain of thought reasoning for selecting plan/action.\nAuthors try both Zero shot and One shot prompts of the above variants.\n\nThe main contribution of the paper is a systematic evaluation of these 3 proactive capabilities. Authors evaluate each task with automatic evaluation metrics as well as human judgments.\n\nFor Clarification dialogues, ProCOT prompting works reasonably well but there is still a large gap to SOTA performance.\nFor Target-guided dialogues, most LLM prompts are good at identifying the next topic. ProCOT strategy also produces smoother topic shift responses.\nFor Non-collaborative dialogues, LLMs struggle in identifying negotiation strategy & dialogue act and consequently the final response is also suboptimal.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Appendix B is crucial for understanding what evaluation metrics were used. These are basically the column headers for many tables in the paper. Without completely understanding those it’s hard to understand the conclusions drawn in the paper. Crucial material like this should not be pushed to the appendix.\n\nIn contrast, there are 6 different values for predicting negotiation strategies and 6 different values for predicting dialogue acts, which one should the reader pay attention to? This is too much information. Most of these numbers will be highly correlated with each other anyway. Is there anything surprising here that it needs to be part of the main paper? It’s not talked about in the main paper, so this can be pushed to the appendix.\n\n\nLine 173, What is a PLM?\n"}}, {"review_id": "NwAKhSAIPb", "reviewer": "Reviewer_e85V", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed and valuable comments. We will carefully address your concerns one by one as follows:\n\n1. \"The paper is very dense and it requires quite some effort to reconstruct what has been done\"\n\n**Response**: We apologize that we have to condense all the content into 8 pages due to the space limitation. However, we believe that this issue can be easily and effectively adjusted by some reorganization. For example, as mentioned by the suggestion from Reviewer NW9t, we can abbreviate some less important evaluation results (e.g., the automatic evaluation results of non-collaborative dialogues) and put the full version into the Appendix, so that we could move the description of the adopted evaluation metrics to the main body. This will largely improve the presentation issue. \n\n2. \"the evaluation results show a rather mixed picture and it is hard to draw general conclusions\"\n\n**Response**:  We did not claim to propose a “SOTA” method or aim to draw the same conclusions for all the problems. The target of this work is to (1) provide a comprehensive evaluation on how the current LLM-based dialogue systems perform in different proactive dialogue problems and (2) testify the effectiveness of different prompting schemes on triggering the proactivity of LLM-based dialogue systems. Thanks to the mixed picture of evaluation results, we could identify what kinds of proactive dialogue problems are difficult to be handled by LLM-based dialogue systems and which aspects of proactivity are lacking in current LLM-based dialogue systems. In order to make the take-aways clearer, we will add a road map to summarize the main findings for each prompting scheme and each proactive dialogue problem. This will be really beneficial for future studies on proactive dialogue systems in this era of LLMs. \n\n3. “A motivation should be given why you test in particular with ChatGPT and Vicuna, and not with other LLMs.” \n\n**Response**:  We appreciate the suggestion. As for the motivation, we would like to evaluate both commercial black-box and open-source LLM-based dialogue systems. Before the EMNLP submission deadline, ChatGPT and Vicuna are the most representative LLMs that are trained for chatting in these two categories, respectively. We will make this motivation clearly discussed. \n\n4. “PLM → LLM”\n\n**Response**:  Thanks so much for pointing out the typo. \n\n5. \"Table 1: hard to reconstruct where the Baseline and SOTA figures come from\"\n\n**Response**:  We apologize for the confusion. We will add more clear descriptions in the table caption. Besides, thanks so much for pointing out the typo of BLEU-2 in Appendix B. \n\n6. \"Line 296: provide details regarding the Help. score\"\n\n**Response**:  We conduct the similar human evaluation as other two tasks. Three annotators are asked to score whether the generated question is helpful for clarifying the existing ambiguity by 0 or 1. The Fleiss’ kappa of the annotations is 0.64, which indicates “substantial agreement”, and the final scores are calculated by average. We will make the human evaluation of the Help. score more clearly. \n\n7. \"Line 371 and Table 3: use of different terminology – next topic selection vs next topic prediction\"\n\n**Response**:  Thanks so much for the suggestion of sticking to the same terminology of “next topic prediction”. We will revise all these terms to be “next topic prediction”.\n\n8. \"Line 388: COLOR is mentioned as one of the fine-tuned baselines for target-guided dialogues, however, it does not show in Table 3, briefly mention why.\"\n\n**Response**: Sorry for the confusion. In the current tables, we just present the reported experimental results from the original paper of each method. Since the model COLOR has not been implemented for the OTTers dataset, we did not present this result as this model was just released in June. In order to clarify the concern, we further implement the COLOR for the OTTers dataset with the open-source code. The results are as follows: \n\n|Method| Shot| Prompt| BLEU| METEOR| R-L| hits@1| hits@3|\n|---|---|---|---|---|---|---|---|\n|TopKG| -| -| 15.35| 13.41| 27.16| 7.78| 22.06|\n|COLOR|-|-|14.53| 13.70| 26.92| 8.02| 22.30|\n\n9. \"Line 402: clarify what R-L stands for: ROUGE-L\"\n\n**Response**: We will clarify that R-L stands for Rouge-L.\n\n10. \"Line 405-407 is not fully backed by the results reported in Table 3 where one-shot results for hits@1 are typically better than zero-shot results.\"\n\n**Response**: Sorry for the confusion. In Line 405-407, we would like to present the fact that the one-shot ICL cannot contribute to a consistent performance improvement, while it may even cause some decreases in certain metrics. We will carefully modify the argument to make this point clearer. \n\n11. \"Line 410-425: also, here it seems the arguments are not fully reflected in the figures presented in Table 3.\"\n\n**Response**: We apologize for any confusion in this discussion. In order to make the statement clearer, Line 419-425 will be revised to be *“With one-shot demonstrations, ChatGPT with proactive prompts continues to underperform compared to standard prompts when it comes to accurately predicting suitable topics towards the target. However, it is worth noting that only ProCoT prompts consistently show an improvement in the performance of all LLMs for next topic prediction.”*\n\n12. \"Table 5: It needs to be explained why TopKG is the only finetuned system used for comparison with the LLMs?\"\n\n**Response**: We would like to clarify that the goal is to compare the LLMs with the SOTA finetuned system. The latest SOTA, i.e., COLOR, was released in June, which was around the submission deadline of EMNLP. Therefore, in our main experiment, we still adopted TopKG as the SOTA finetuned system for experiments. In order to present an up-to-date result, we further conduct the same human evaluation on the generated dialogue from COLOR. The updated Table 5 is as follows:\n\n| |  | Easy | Target | Hard | Target| \n|---|---|---|---|---|---|\n|Method| Prompt| G-Coh.| Effect.| G-Coh.| Effect. | \n|TopKG| -| 1.42| 1.24| 1.21| 1.10|\n|COLOR|-| 1.38| 1.42 | 1.24 | 1.18 |\n\n13. \"All tables: make sure to provide all relevant information in the caption, e.g., for the measures what is good – high or low value; in Tables 3 and 4, mention which corpora are used; Table 4 and 5, briefly describe what is a hard/easy target.\"\n\n**Response**: Sorry for the confusion regarding this information. Due to the space limitation, we did not re-emphasize this information in the table caption. The used corpora in Tables 3 and 4 is OTTers and TGConv, respectively, as introduced in Line 390-391 and Line 427-428. Hard/easy target represents the hard-to-reach and the easy-to-reach target in the TGConv dataset, as introduced in Appendix A. We greatly appreciate the thoughtful and valuable suggestions regarding elaborating more details in the table captions. We will carefully add more detailed and relevant information to make the table clearer. \n\n14. \"there are a few issues with subject verb congruency in the text.\"\n\n**Response**: We will also carefully review the whole paper to address them. \n\nOverall, as the main concern is regarding the dense presentation due to the space limitation, we believe that we can effectively address this concern by taking into account your valuable suggestions. Again, thanks so much for the detailed reviews.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Broad investigation of LLMs’ (ChatGPT, Vicuna) proactive dialogue capabilities under different prompting schemes and under zero-shot and one-shot conditions, as well as their comparison with state-of-the-art fine-tuned models. \nThe work offers a wealth of material and references related to proactive dialogue systems and their evaluation.\n", "reasons_to_reject": "The paper is very dense and it requires quite some effort to reconstruct what has been done, e.g., it is not always clear what is in the tables, where the baseline and SOTA figures come from, how hard and easy targets in dialogue-level evaluation are defined, etc. (see Presentation Improvements). More streamlining and guidance would be required to make the paper ready for publishing in a conference. Moreover, the evaluation results show a rather mixed picture and it is hard to draw general conclusions. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents an analysis of LLMs (in particular ChatGPT and Vicuna) regarding their capabilities to generate proactive dialogue under different prompting schemes – standard, proactive and proactive chain of thought (ProCoT) and under zero-shot and one-shot conditions. ProCoT is defined by the authors to combine CoT with proactive strategies. The following three proactive dialogue capabilities are investigated: clarification dialogues in information seeking, proactively leading the conversation towards a designated target, and reaching a consensus with the user when user and system do not have the same conversational goal. The results are compared with results from fine-tuned models (baseline and SOTA) and the outcomes are discussed. The presented work makes use of a variety of methods and evaluation metrics. Which is an asset! The analyses result in a broad description of (mostly missing/weak) proactive dialogue capabilities of the two tested LLMs under the respective prompting schemes and 0/1-shot conditions. In other words, the work provides a picture of what is, but less so why.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "A motivation should be given why you test in particular with ChatGPT and Vicuna, and not with other LLMs.\n\nLine 173: PLM ?  LLM\n\nTable 1: hard to reconstruct where the Baseline and SOTA figures come from (except for Abg-CoQA BLEU-1), clarify for all Baseline and SOTA figures which models were used and where exactly (reference plus where in the reference) the individual figures are published. Note there is also an inconsistency between Table 1 (BLEU-1) and Appendix B.1 which says BLEU-2 is used. \n\nLine 296: provide details regarding the Help. score, such as how many annotators, what are the criteria used, what is the intercoder agreement.\n\nLine 371 and Table 3: use of different terminology – next topic selection vs next topic prediction; this might be confusing for the reader, so stick to one term throughout the paper.\n\nLine 388: COLOR is mentioned as one of the fine-tuned baselines for target-guided dialogues, however, it does not show in Table 3, briefly mention why. \n\nLine 402: clarify what R-L stands for: ROUGE-L\n\nLine 405-407 is not fully backed by the results reported in Table 3 where one-shot results for hits@1 are typically better than zero-shot results.\n\nLine 410-425: also, here it seems the arguments are not fully reflected in the figures presented in Table 3.\n\nTable 5: It needs to be explained why TopKG is the only finetuned system used for comparison with the LLMs?\n\nAll tables: make sure to provide all relevant information in the caption, e.g., for the measures what is good – high or low value; in Tables 3 and 4, mention which corpora are used; Table 4 and 5, briefly describe what is a hard/easy target. \n\nNote, there are a few issues with subject verb congruency in the text.\n"}}, {"review_id": "fVtU310ics", "reviewer": "Reviewer_eUzu", "summary": "", "questions": "No questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your acknowledgement of our work. We believe that this work can provide a comprehensive picture about how the current LLM-based dialogue systems can handle different types of proactive dialogue systems. It also discusses which aspects of the proactivity can be improved for further enhancing the LLM-based dialogue systems for proactive dialogues. \n\nRegarding the materials in Appendix, we greatly appreciate the suggestions. We will carefully reorganize the space. For example, we will move Appendix B to the main body by abbreviating the automatic evaluation results of non-collaborative dialogues as well as putting its full version into the Appendix. We strongly agree that this will make the analysis clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main findings of the evaluation of LLM-based dialogue systems are explained:\n- They barely ask clarification questions when encountering ambiguous queries;\n- They tend to make aggressive topic transition. \n- They fail to make strategic decisions.\nIt is shown how the proposed ProCoT scheme can help to overcome these limitations.\n\n", "reasons_to_reject": "The paper should be accepted.", "questions_to_authors": "No questions.", "paper_topic_and_main_contributions": "The paper introduces the first comprehensive analysis of Large Language Model-based conversational systems on three common aspects of proactive dialogues (clarification, target-guided, and non-collaborative dialogues).\n\nMain contributions:\n-The proactive chain-of-thought prompting scheme (ProCoT) is designed to endow the dialogue systems with the capability of planning and taking the initiative towards the conversational goal.\n-Experiments with two LLM-based conversational systems, including ChatGPT and an open-sourced model (Vicuna) are carried out, the findings are analyzed.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing references.", "Typos_Grammar_Style_and_Presentation_Improvements": "No typos.\nHowever, the Appendix (pages 12 to 20, starting from line 962) includes the material that actually belongs to the main part of the paper (still, page limit is 8). "}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an exploration of proactive dialogue systems using prompted LLMs, with a systematic evaluation of their capabilities. While the paper makes interesting contributions, there is a lack of reproducibility of the results since they rely on very specific prompting techniques.
Decision: Accept-Findings