Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curation
Authors: Jian Wang
Yi Cheng
Dongding Lin
Chak Tou Leong
Wenjie Li
Venue: EMNLP 2023 Main (2023)
Keywords: Target-oriented dialogue
Proactive dialogue
Personalization
Dataset curation

Abstract: Target-oriented dialogue systems, designed to proactively steer conversations toward predefined targets or accomplish specific system-side goals, are an exciting area in conversational AI. In this work, by formulating a <dialogue act, topic> pair as the conversation target, we explore a novel problem of personalized target-oriented dialogue by considering personalization during the target accomplishment process. However, there remains an emergent need for high-quality datasets, and building one from scratch requires tremendous human effort. To address this, we propose an automatic dataset curation framework using a role-playing approach. Based on this framework, we construct a large-scale personalized target-oriented dialogue dataset, TopDial, which comprises about 18K multi-turn dialogues. The experimental results show that this dataset is of high quality and could contribute to exploring personalized target-oriented dialogue.

Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curation
Jian Wang, Yi Cheng, Dongding Lin, Chak Tou Leong, Wenjie Li Department of Computing, The Hong Kong Polytechnic University (jian-dylan. wang, alyssa. Mhyssa,cheng, dongding88 1in, chak-tou. ceonntou.leong) @connect. cconnect.polyu.... hk cswjli@comp. wtw.iieccopppplyuu.. edu.
Abstract
of Comedy">, the system needs to take appropri- ate dialogue acts and smoothly steer the discussed topic towards the designated onc. Its ultimate ob- jective is to achieve recommendations on the target topic "King of Comedy". Our work also follows the form of <dialogue act, topic> pairs as targets to study target-oriented dialogue systems due to their higher applicability in real-world scenarios.
Target-oriented dialogue systems, designed to proactively steer conversations toward prede- fined targets accomplish specific system-side goals, are an exciting area in conversational Al. this work, by formulating <dialogue act, topic> pair as the conversation target, we ex- plore a novel problem of personalized target- oriented dialogue by considering personaliza- tion during the target accomplishment process. However, there remains an emergent need for high-quality datasets, and building one from scratch requires tremendous human effort. To address this, we propose an automatic dataset curation framework using role-playing ap- proach. Based on this framework, we construct a large-scale personalized target-oriented dia- logue dataset, TOPDIAL! which comprises about 18K multi-turn dialogues. The experi- mental results show that this dataset is of high quality and could contribute to exploring per- sonalized torget-oriented dialogue.
Despite many existing efforts, we find that two critical issues remain be solved. One urgent problem is the need for well-organized benchmarks or datasets. Current studies for target-oriented di- alogue (Gupta et al., 2022; Wang et al., 2023a) mainly re- re-purpose existing non non-target-oriente di- alogue datasets, which are not exactly suitable as they are crowd-sourced without consideration of target accomplishment. Nevertheless, building a new high-quality dataset from seratch requires pensive human effort. The other essential issue is that, larget-orienter dialogue systems need con- sider personalized aspects (Wu et al., 2021: Rana et al., 2023), such as user profiles and personali- ties, which were largely ignored by previous work. User profiles involve user preferences about poten- tial topics relevant to the target, while personalities imply possible reactions and feedback during the dialogue process. With personalized information incorporated, the system could be tailored to a user and lead the conversation towards the target with higher engagement instead of obtrusively driving to the target, thereby improving user experience. Thus, we raise the question: How can we build high-quality datasets with little human effort for personalized target-oriented dialogue?
1 Introduction
Compared with traditional dialogue systems that fo- cus merely on passively responding to user require- ments, a recently investigated research topic of target-oriented dialogue systems (Sevegnani et al., 2021; Deng et al., 2023) specifies conversation target from the system side, enabling the system to take the initiative and lead the conversation. Early work in this mainly formulates the targets as mentioning certain keywords (Tang al.. 2019; Qin al., 2020; Zhong et al., 2021; Yang et al., 2022) or specific topics (Wu al., 2019; Seveg- nani et al., 2021). To allow the formed targets to be applicable in broad scenarios, few recent stud- ies (Zhang et al., 2021; Wang et al., 2023b) define <dialogue act, topic> pairs as targets. For example, given the target of <movie recommendation, "King
In this work, we first give comprehensive defi- nition ($2) of personalized target-oriented dialogue, then lay out the desirable characteristies ($2) that qualified dialogue dataset should meet. Draw- ing inspiration from some recent work that has demonstrated unprecedented capabilities of large language models (LLM) in simulating human so-
'Our code and data are available at https://github. com/iwangjian/ ia.laa

---

cial behaviors (Guo al., 2023; Li al., 2023), we propose a role-playing approach for automatic dataset curation ($3) using multiple LLM agents, They are designed to follow specific instructions to fulfill the requirements. Based on that, we syn- thesize a large-scale dialogue dataset named TOP- DIAL and show its quality and effectiveness ($4).
Role-Piaying Environment
Tarper lest
System
Seed Dorasct
Bonsin Krowlcage
Prompting
Bull Buheagment
Moderator
S
trarage sugus
Di
LLM
Our main contributions are: (1) formulate the problem of personalized target-oriented dialogue, which is promising yet underexplored. (2) We pro- pose novel role-playing framework for automatic dialogue dataset curation. It provides insights into building large-scale datasets for many other dia- logue tasks. (3) Our constructed TOPDIAL dataset is of high quality and contributes the related research community.
User
die [
Sampling
Samulinee Uber Phofie
Phonpting
Big Trahh
Figure 1: Overview of our role- role-nlaying framework for automatic dialogue dataset curation.
Dataset Curation Framework
2 Problem Formulation
In this section, we describe a role-playing approach for automatic dataset curation using multiple LLM agents. Figure/ depicts the whole framework, which involves one user agent, one system agent, and one moderator agent. All these agents are designed to follow specific instructions and com- municate in our role-playing environment.
Task Definition We consider dialogue corpus = {lu K,, Tr where N the total number of dialogues. In the i-th dialogue, U, rep- resents the personalized information, such as the user's profiles and/or personalities. K;; represents the domain knowledge facts relevant to the i-th di- alogue. T; denotes the predefined target consisting of <dialogue act, topic> pair. C, ( is the dialogue content, with a total of Nr turns. The task of personalized target-oriented dialogue is formalized as follows: given a target T, set of user's personalized information 2, a set of relevant domain knowledge X. and dialogue context C, the objective is to proactively lead the conversation and generate proper utterances to achieve the target T at appropriate time.
Role-Playing Environment This environment is designed to provide global description for prompt- ing all LLM agents. To achieve desirable target- oriented role playing, we instantiate the environ- ment description based on the domains of the pre- defined targets. For example, one can describe the environment as "You are participating in a conversation about music or movies." for a given target T = <movie recommendation, "King of Comedy">. Then, the description will be prepended to each agent's 's instructions.
Desirable Characteristics of Datasets Based on the above definition, we lay out two desirable characteristics that qualified dataset should meet, namely target-oriented proactivity and personaliza- tion. Target-oriented proactivity emphasizes that a dialogue dataset should allow the system to (i) take the initiative throughout a conversation, (ii) proac- tively lead the discussed topic towards the target topic based on domain knowledge, and (iii) accom- plish the target act. On the other hand, personaliza- tion indicates that dialogues in qualified dataset should embody (i) user profiles, which may involve users' past preferences about potential topics rele- vant to the target, and (ii) user personalities, which may imply users' possible reactions and feedback during the system-initiative process.
User Agent The user agent aims to simulate hu- users who generate utterances conditioned on their specific profiles and personalities. Since there are many off-the-shelf dialogue datasets grounded with user profiles, we collect all user profiles from one chosen dataset and parse them into a profile slot pool. Each slot contains a particular slot key (e.g., name, age range, liked or disliked movies) and a list of candidate values. We randomly sam- ple slot value for each key, and then form all key-value pairs as the simulated user profile.
Inspired by Big-5 personality traits (Goldberg, 1993) that have been widely adopted in personality- aware tasks (Oraby et al., 2018; Yu et al., 2019), we randomly sample a positive or negative description

---

Dataset
Participants
Formed Targets
TO
PF
PN
Domaius
MT
#Dialogue
TGC (Tang et al., 2019)
Crowd workers
Keywords
/
X
Open-domain Movics
9,939
DuConv (Wu et al., 2019)
Crowd workers
Topical catities
/
X
29,858
TG-ReDial (Zhou al. 2020)
Crowd workers
N/A
K
V
Movics
10,000
OTTers (Sevegnani al., 2021)
Crowd workers
Topics
v
X
Open-domain
X
4,316
IGConv (Yang ct al., 2022)
Crowd workers
Keywords
V
X
X
Open-domain
18,878
DuReeDial 2.0 (Liu ct al., 2021)
Crowd workers
N/A
X
V
Movics, music, food, POLs"
16,482
DuRecDial 2.0" (Wang et al., 2023a)
Human experts
Act-topic puirs
Movies, music, food, POLs"
6,080
TOPDIAL (Ouns)
LJM ugents
Act-topic pairs
Movies, music, lood, POIs
18,009
Table Comparison between TOPDIAL. and other related datasets (TO: target-oriented, PF; profile grounding, PN: personality grounding. MT: multi-tum conversation, t: re-purposed version, *: point-of-interest restaurants).
for each of the following traits: openness (O), con- scientiousnesss (C), extraversion (E), agreeableness (A), neuroticism (N). The sampled descriptions are then combined as the simulated user personality. We verbalize the simulated user profile and person- ality in natural languages, prompting the user agent act as human user. We present our detailed instruction template Appendix A.I.
Total dialogues (train test) ulterances valid test)
12,601 1,802 3,666
141,928 20,310
Total targets
501
Avg. slot keys per user profile
Avg. traits per personality
Avg. knowledge triples per dialogue
36.8
Avg. utterances per dialogue
12.3
Avg. # words user's tam
28.2
Avg. # words per system's 's turn
37.5
Table 2: Statistics of the TOPDIAL dataset.
System Agent The system agent aims to as human-like domain-specific enthusiast, such as a movie enthusiast who enjoys variety of films, or a foodie who enjoys delicious food. Its long- term goal is to proactively lead the conversation towards the target, as discussed in $2. To achieve target-oriented proactivity, we take a given target T and set of relevant domain knowledge K. (and few comments related to the target topic, if applica- ble) from chosen seed dataset as the fundamental prompting source. Besides, in conversations, one can easily know the other's ex- plicit profile information, while hard to be aware of implicit personality before their first con- versation. Thus, we pass the simulated user profile yielded by the user agent to the system agent as personalized prompting source (see Figure 1).
ohinterests (POls)
Food
Movies
43%
Music
Figure 2: Distribution of domains among dialogues.
tions are outlined follows: (1) The system agent completes the target act (e.g., recommendation) on the target topic, the user agent accepts and the system no longer takes the initiative for two rounds. (2) The user agent explicitly rejects the system agent's s act on the target topic for the sec- ond time. (3) The conversation between the system and the user agents reaches a maximum number of rounds. For the first two conditions, we take few dialogues from the seed dataset as in-context amples to demonstrate whether not ongoing conversation should be terminated. We present the detailed instruction template Appendix A.3.
We assign required instructions to the system agent based on the above prompting sources and task definition. We provide the instruction template in Appendix A.2. In practice, we further enhance the system agent in self-augmented instruction manner, where the agent's task prompt will be re- peated at each dialogue round avoid forgetting its long-term goal.
Dataset Curation We employ three ChatGPT
The moderator agent is de- signed to automatically manage the termination of the conversation between the system and the user agents. To ensure that the synthetic data adhere to desirable characteristics, we set certain condi- tions to terminate the conversation. These condi-
Moderator Agent
(gpt-3.55 3.t-rrbbb version) agents LLM agents for the above roles. We ask the system agent initiate a greeting with the user agent, and they will chat turn by turn, resulting in multi-turn conversa- tions. Their conversations are terminated by the moderator agent or the maximum limit of rounds.

---

Recerment
Ekt Parzd
Racamment
Chat sbou sare
Reepped GAART
sbout
Recverend PCBu
sbout
di HeteerG
Recommars Poress
Rocartical lood
Hoovmnond
Figure Transitions dialogue acts of the system through the first rounds.
The three agents can generate large-scale dialogues through their collaboration, with very little human effort involved in the whole process.
Win
Tle
Succ:
Pas..
Human
60
15
CheedPT
20
4 TOPDIAL Dataset
Proace Coh.
Humana
ChatGPT
Based on our dataset curation framework, we syn- thesized the dataset TOPDIal by utilizing the re- purposed version (Wang et al., 2023a) of DuRec- Dial 2.0 (Liu et al., 2021) as the seed dataset after carefully considering the problem formulation and necessary prompting sources. We report more im- plementation details in Appendix B.I.
Human
51
ChetSPT
52
0%
20%
40%
60%
80%
100%
Figure Automatic and human evaluation results be- tween the seed dataset and ours (TOPDIaL).
Automatic and Human Evaluations To assess the quality of TOPDIAL, we conduct LLM-based automatic evaluation and human evaluation. We randomly choose 100 targets and then sample one dialogue per target from the seed and TOPDIAL datasets, respectively. We ask ChatGPT (Ope- nAI, 2022) and human evaluators compare each pair of dialogues over four metries: proactivity (Proact.), coherence (Coh.), personalization (Pers.), and target success rate (Succ.). We provide de- tails for these metrics and our evaluation settings in Appendix B.2.
Dataset Statisties Table compares TOPDIAL with related datasets. To the best our knowledge, TOPDIAL is the first dataset equipped with the de- sirable characteristics discussed in $2. should be noted that the DuRecDial 2.0 dataset crowd- sourced without considering targets and is not ex- actly suitable for the end task of target-oriented proactive dialogue, while the re-purposed version of DuRecDial 2.0 largely relies on human effort form targets and preprocess dialogues. In com- parison, our TOPDIAL. dataset is curated based on target-oriented proactivity. addition, by ground- ing the personality information during the dataset curation process, TOPDIAL is more natural and effective in reflecting personalization.
Figure 4 shows the evaluation results, where Fleiss's 's kappa (Fleiss, 1971) scores are distributed between [0.41, 0.60J, indicating moderate inter- evaluator agreement. We observe that for all met- rics, the TOPDIAL dataset achieves comparable and slightly higher win percentages over the seed dataset. verifies the high quality of TOPDIAL.
Table 2 shows detailed statistics of the TOPDIAL dataset (see domain distributions in Figure 2). We also visualize the transitions of dialogue acts of the system through the first six dialogue rounds in Figure 3. We observe that the system often asks preferences other questions the very beginning. As the dialogue continues, the system introduces topic-related attributes and elicits the user's interest, It shows that the system proactively leads the dia- logue and gradually achieves target dialogue acts, i.e., recommendations on target topics.
Dataset Evaluation by Baseline Models We quantitatively evaluate TOPDIAL using representa- tive dialogue models, including DialoGPT (Zhang al., 2020) and Alpaca-7B (Taori al., 2023). We fine-tune these models the seed and TOPDIAL datasets, respectively, with an identical training data size. For fair comparison, we build the test set for evaluation with 50% from the seed test data

---

Avg. BLEU
Knowledge F1 (%)
Persoma F1 (%)
process is challenging, ChatGPT may produce oulput content with hallucinations (Bang al., 2023). We intend to improve the dataset curation process with some protessoess steps, such as fact-checking and correction based on the grounded domain knowledge. In addition, we observe that sometimes the moderator agent cannot appropri- ately terminate conversation due to its difficullyy in understanding the achievement of the target, even though has been assigned with detailed instruc- tions and in-context examples. We will leave this for future research.
Model
Succ. (%)
DialoGPT w/S 0.127
24.62
21.55
32.94
DialoGPT w/
0.138
47.42
30.51
51.83
Alpaca TB w/ 0.177
38.60
37.05
48,78
Alpaca-78 w/' 0.229
57.12
51.99
85.04
Table 3: Performance of baseline models trained on the
seed (S) dataset and our TOPDIAL. (T) dataset.
and 50% from the TOPDIAL test data. Our evalu- ation metrics include the average score of BLEU- 1/2 (Papineni et al., 2002), persona F1 (Lim et al., 2022), knowledge F1 and target success rate (Succ.) (Wang et al., 2023a). We deseribe details of these metrics and model training in Appendix C.
Ethical Considerations
Developing target-oriented dialogue systems quires careful ethical considerations due to the po- tential impact on specific scenarios. As an appli- cation scenario explored this work, providing recommendations is one of the highly-applicable target dialogue acts. Target-oriented dialogue sys- tems can create non-obtrusive recommendations for specific products and services. Our work does not force the system to achieve the designated tar- get nor force users accept recommendations.
The comparison results reported in Table 3 show a similar trend: the two baseline models trained our TOPDIAL dataset significantly outperform those trained on the seed dataset. In particular, our TOPDIAL dataset more effective in train- ing personalized target-oriented dialogue models (e.g., much higher persona F1 and Succ. socres) by grounding the profile and personality information during the dataset curation process. It shows that TOPDIAL is an effective training resource for the personalized target-oriented dialogue task.
We emphasize that regulation of the target des- ignation is crucial when deploying target-oriented dialogue systems in particular domains. For in- stance, specifying a target should not violate factual correctness, privacy rules, laws of human society. We want to raise awareness about the po- tential misuse of such systems with toxic intentions. For example, such systems may be used to pose as humans and mislead users through conversations. To avoid such risks, we highlight that it is necessary to improve transparency, such as informing users that they are chatting with bot, not a human.
Case Study Due to space limitation, we present some cases in Appendix D (see Figure 9 and Fig- ure 10) for better understanding. These cases intuitively show that our TOPDIAL dataset fulfills target-oriented proactivity and personalization. also shows that our dataset curation framework can be a viable alternative for building personalized target-oriented dialogue datasets.
5 Conclusion
In this work, we explore task: personalized target-oriented dialogue. We first define this chal- lenging task, and then lay out the desirable char- acteristics that a qualified dialogue dataset should meet. We propose novel role-playing framework for automatic dataset curation, based on which we construct a large-scale dialogue dataset TOPDIAL. Our statistics and evaluations validate its effective- ness and high quality.
Acknowledgments
This work was supported by the Research Grants Council of Hong Kong (15207122, 15207920, 15207821, 15204018, 15213323) and National Nat- ural Science Foundation of China (62076212). It was also supported in part by PolyU internal grants (ZVQ0, ZVVX).
Limitations
References
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung. et 2023. A multi- task, multilingual, multimodal evaluation of chatgpt
Since we adopt ChatGPT agents to simulate the designed roles, ensuring the factual correctness the synthetic dialogues during the role-playing

---

on reasoning, hallucination, and interactivity, arXiv preprint arXhv:2302.04023.
Jungwoo Lim, Myunghoon Kang, Yuna Hur, Se- ung Won Jeong, Jinsung Kim, Yoonna Jang. Dongyub Lee, Hyesung Ji, DongHoon Shin, Seun- gryong Kim, and Heuiseok Lim. 2022. You truly un- derstand what need: : Intellectual and friendly dialog agents grounding persona and knowledge. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 1053- 1066, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Yang Deng, Wenqiang Lei, Wai Lam, and Tat-Seng Chua. 2023. A survey on proactive dialogue systems: Problems, methods, and prospects. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 6583-6591. International Joint Conferences on Artificial Intelli- gence Organization. Survey Track.
Emily Dinan, Angela Fan, Adina Williams, Jack Ur- banek, Douwe Kiela, and Jason Weston. 2020.
Zeming Liu, Haifeng Wang, Zheng- Yu Niu, Hua Wu, and Wanxiang Che. 2021. DuRecDial 2.0: A bilin- gual parallel corpus for conversational recommen- dation. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 4335-4347, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Queens are powerful too: Mitigating gender bias in dialogue generation. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 8173-81 8188, Online. As- sociation for Computational Linguistics.
Joseph I. Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
OpenAL. 2022. Introducing ChatGPT. https:// // openai wwt.ccowm/llccecccecc com/blog/chattpt..
Lewis Goldberg, 1993. The structure of phenotypic personality traits. American psychologist, 48( 1):26.
Shereen Oraby, Lena Reed, Shubhangi Tandon, Sharath T.S., Stephanic Lukin, and Marilyn Walker. 2018. Controlling personality-based stylistic variation with neural natural language generators. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 180-190, Melbourne, Australia. Association for Computational Linguisties.
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arxiv:2301.07597
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318. Philadelphia, Pennsylvania, USA. Association for Computational Linguisties.
Prakhar Gupta, Harsh Jhamtani, and Jeffrey Bigham. 2022. Target-guided dialogue response generation using commonsense and data augmentation. In Find- ings of the Association for Computational Linguis- tics: NAACL 2022, pages 1301-1317, Seattle, United States, Association for Computational Linguistics.
Jinghui Qin. Zheng Yc, Jianheng Tang, and Xiaodan Liang. 2020. Dynamic knowledge routing network for target-guided open-domain conversation. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 05, pages 1657-8664.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. 2022. Lora: Low-rank adaptation of large lan- guage models. In International Conference on Learn- ing Representations.
Arpit Rana, Scott Sanner, Mohamed Reda Bouad- jenek, Ron Dicarlantonio, and Gary Farmaner. 2023. User experience and the role of personalization in critiquing- based conversational recommendation. ACM Transactions on the Web.
Minju Kim, Chachyeong Kim, Yong Ho Song. Seung- won Hwang, and Jinyoung Yeo. 2022. BotsTalk: Machine-sourced framework for automatic curation of large-scale multi-skill dialogue datasets. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 5149- 5170, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Karin Sevegnani, David M. Howcroft, Toannis Konstas, and Verena Rieser. 2021. OTTers: One-turn topic transitions for open-domain dialogue. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2492-2504, Online. Association for Computational Linguisties.
Guohao Li, Hasan Abed AI Kader Hammoud, Hami Itani, Dmitrii Khizbullin, and Beraad Ghanem. 2023. Camel: Communicative agents for "mind" explo- ration of large scale language model society. arXiv preprint 2:.3:::333.77760
Jianheng Tang. Tiancheng Zhao, Chenyan Xiong, Xiao- dan Liang, Eric Xing, and Zhiting Hu. 2019. Target- guided open-domain conversation. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 5624-5622 Florence, Italy. Association for Computational Linguistics.
Margaret Li, Jason Weston, and Stephen Roller. 2019, Acute-eval: Improved dialogue evaluation with op- timized questions and multi-turn comparisons. Ad- vances in Neural Information Processing Systems, Conversational AI Workshop.

---

Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction- fooloooinn llama model. https:// gi github. com/
Liu, and Bill Dolan. 2020. DIALOGPT: : Large-scale generative pre- pre-training for conversational response generation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online. As- sociation for Computational Linguisties.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971.
Hanxun Zhong, Zhicheng Dou, Yutao Zhu, Hongjin Qian, and Ji-Rong Wen. 2022. Less more: Learn- ing to refine dialogue history for personalized dia- logue generation. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pages 5808-5820, Seattle, United States. Association for Computational Lin- guistics.
Jian Wang, Dongding Lin, and Wenjie Li. 2023a. Di- alogue planning via brownian bridge stochastic pro- cess for goal-directed proactive dialogue. In Find- ings of the Association for Computational Linguis- tics: ACL 2023, Toronto, Canada. Association for Computational Linguistics.
Peixiang Zhong, Yong Liu, Hao Wang, and Chunyan Miao. 2021. Keyword-guided neural conversational model. In Proceedings of the AAAI Conference on Artificial Intelligence, 16, pages 145661
Jian Wang, Dongding Lin, and Wenjie Li. 2023b, A target-driven planning approach for goal-directed dia- log systems. IEEE Transactions on Neural Networks and Learning Systems.
Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji- Ji-Rong Wen. 2020. Towards topic-guided conversational recommender system. In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 4128-4139, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang- 2019. Preactive buman-machen machine conversation with explicit conversation goal. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 3794-3804, Florence, Italy. Association for Computational Linguistics.
A Instructions for Different Agents
Yuwei Wu, Xuezhe Ma, and Diyi Yang. 2021. Personal- ized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1970, Online. Association for Computational Linguistics.
A.1 User Agent
We provide the assigned instruction template for the user agent in Figure 5.
A.2 System Agent
We provide the assigned instruction template for the system agent in Figure 6.
Zhitong Yang, Bo Wang, Jinfeng Zhou, Yue Tan, Dong- ming Zhao, Kun Huang, Ruifang He, and Yuex- ian Hou. 2022. TopKG: Target-oriented dialog via global planning on knowledge graph. Proceed- ings of the 29th International Conference on Com- putational Linguistics. pages 745-755, Gycongju, Republic of Korea, International Committee on Com- putational Linguistics.
A.3 Moderator Agent
We provide the assigned instruction template for the moderator agent in Figure 7.
B TOPDIAL Dataset
Mingzhi Yu, Emer Gilmartin, and Diane Litman. 2019. Identifying personality traits using overlap dynamics in multiparty dialogue. Proceedings of Interspeech 2019, pages 1921-1923
B.1 Implementation Details of Dataset Curation
In this work, we implemented our role-playing framework based on an open-source library named ChatArena?. We called the gpt-3. 5-trrbbo ver- sion of ChatGPT API to build each LLM agent. We adopted a temperature of 0.75 to generate re- sponses for all agents. We set the maximum num- ber of tokens to generate to 100, 80, and 20 for
Jun Zhang. Yan Yang. Chencai Chen, Liang He, and Zhou Yu. 2021. KERS: A Enowledge-enhanced framework for recommendation dialog systems with multiple subgoals. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1092-1101, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Yizhe Zhang. Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing
hnttps://plaffome wpwnai.coomdocss....... com/docs/ api-refference/chat

---

User Profile-specific Prompt:
You are NAME>, male/female student the age range RANGE>, living in <RESIDENCE> man/woman in the age range of LAGE_RANGE> working a company and living in <RESIDENCE> retired man/woman the age range <AGE_ RANGE>, living in <RESIDENCE>> Based on your past experiences, you have the following preferences: Your liked <SYOT_KEY>: -SLOT_VALUE>
Your disliked <SLO;_KEY>; <SLOT_VALUE
User Personality-speeifte Prompt:
Based on the Big-5 personality traits, your personality measured as:
For openness, you are (intellectual, imaginative, and curious unimaginative, uncreative, and conventional). For conscientiousness, you are (efficient, organized, and careful inefficient, careless, and sloppy)
For extraversion, you are (outgoing, energetic, and talkative I shy, reserved, and quiet). For agreeableness, you are (trustworthy, straightforward, and generous unreliable, complicated, meager, and boastful)
For neuroticism, you are (sensitive, nervous, and insecure secure, confident, and calm).
Task Prompt:
Your response should be concise (no longer than 30 words).
You don't need recommend anything, but feel free to express your personal interests. You don't need prepend your to your response, despite others may do it.
Figure 5: Instruction template the user agent. This involves the user profile-specifee prompt, user personality- specific prompt, and task prompt.
System Role Prompt:
You WNSTEM_NAMES movie enthusiast who enjoys variety of films | music enthusiast who enjoys variety of music foodie who enjoys delicious food food enthusiast who is interested in exploring different restaurants.
User Profile-specific Prompt:
You are conversing with <USER_NAME>, whose profile below: #* <USER_PROFILES
Task Prompt:
Your goal is to proactively lead the conversation with <USER_NAME> towards the target (movie music food point-of-interest, POI) <TARGET_TOPIC>. To start the conversation, please begin with a greeting and avoid mentioning the target (movie music food | POI).
As the conversation progresses, use your domain knowledge to steer the topic threads towards the target (movie | music | food POI) step by step.
Be informative and engaging while providing insights to arouse <USER_NAME>> interest. Remember to ultimately recommend SARGET_TOPICC
the focus of the conversation.
Your words at each turn should be concise (no longer than 30 words).
You may access the following domain knowledge for conversation: ## <DOMATN_KNOWLEDDE_TTTTTT TRIPLES>
Figure 6: Instruction template for the system agent. This involves the system role prompt, user profile-specific prompt, and task prompt.

---

You the moderator of conversation. You need determine whether the discussion between <SYSTEM_ and <USER_NAME> NAME> should came immediate end. The conversation should be terminated under the following two conditions:
(1) <SYSTEM_NAME> NAME>> completes recommendation
<TARGET_TOPIC>
<USER_NAME> accepts it, and
<SYSTEM_NAME no longer takes initiative for two rounds.
(2) If DUSER_NAME>> explicitly rejects SYYSTE recommendation <TARGET_TOPIC> when <SYSTEM_NAMES has tried to recommend for the second time.
In either of these cases, the conversation should be brought to an immediate end.
For example, here is a conversation: Should the conversation end? The answer no. Here is another conversation: ## SSEED_DIALOGUE_2 Should the conversation end? The answer is yes.
1EED_DDALL
Now, for the following conversation: #1l WINGOINGGDIALOGLU
Should the conversation end? Answer yes or no.
Figure 7: Instruction template for the moderator agent. This involves comparative in-context examples improve the instruction.
the system, user, and moderator agents, respec- tively. We set a maximum limit of 8 rounds based on our observation of target accomplishment while ensuring that the dataset curation is not too costly. We synthesized three different dialogue instances for each seed example in the chosen seed dataset, i.e., the repurposed version (Wang et al., 2023a) of DuReeDial 2.0 (Liu et al., 2021). On average, the cost of API calls is approximately 0.032 $ for one dialogue. We obtain two types of splits for the test set: seen and unseen, similar Sevegnani et al. (2021); Wang et al. (2023a). The test-unseen split ensures that none of the target topics in the test set are present in the training set, whereas the test-seen split allows them to appear.
evaluators and ask them to perform blind pair- wise comparison. Specifically, we employ ACUTE- EVAL (Li et al., 2019), widely used dialogue evaluation platform for multi-turn turn dialogue evalu- ation (Dinan et al., 2020; Kim et al., 2022). We adopt Fleiss's 's kappa (Fleiss, 1971) to measure the agreement among the human evaluators. Figure 8 shows the interface used for human evaluation.
We ask ChatGPT and human evaluators com- pare each pair of dialogues terms of the fol- lowing metries: proactivity (Proact.), coherence (Coh.), personalization (Pers.), and target success rate (Succ.), similar to related studies (Wang et al., 2023a; Kim et al., 2022). We use question form to describe these metrics, with the wording of ques- tions presented as follows:
B.2 Settings of Automatic and Human Evaluations
Proactivity (Proact.): Which dialogue shows that the system takes the initiative during the con- versation and proactively leads the topic threads toward the target topic?
We describe the settings for LLM-based automatic evaluation and human evaluation that we conduct validate the quality of the constructed Top- DIAL dataset. We randomly choose 100 targets and then sample one dialogue per target from the seed and TOPDIAL datasets, respectively. We only include the targets and dialogue contexts while excluding grounded contexts (e.g., domain knowledge and personalized user information) for anonymity, since the grounded contexts of the seed and TOPDIAL datasets are distinguishable. For LLM-based automatic evaluation, we employ the gp Surbbo version of ChatGPT compare each pair dialogues. For human evaluation, we recruit three edul-edde graduate students as
Coherence (Coh.): Which dialogue is more nat- ural and coherent, like humans? Whose dialogue context flows more smoothly?
Which dialogue reflects the user's preferences or personalities more? Which dialogue is more likely arouse the user's interest?
Personalization (Pers.):
Target Success Rate (Succ.): Which dialogue successfully achieves the target dialogue act on the target topic?

---

Which Dialogue is Better?
190
your tmaths?
carnert comp partion
Kook tit one corte coacrties not for of orAmartion Draam
Yor rose deltgguss fer pionttgge repermennirees
Erilkeing
"ureey thakkeess jont pagen ben ne yf coneerttin You
Yarrs secone the song trin noon vey the sares aeeg aho gass. it aingin
That coento neart
are ws Jing lent musce
amgar and TV non fon
awarte
You bdgel d 2o
Glu agud Rot tome tise
mencce proadiilly only perto lustin,, lang auscass you
Foe muuing som sant usen mons
In commor wern cang lank bult patrsdey glu
quet pangerarta paga, sutton
Hhat
Loney dhucker
any menly mane se marge are sate comeiie also swarts
The
th live Wounly torite L24 chornd tordtho
Yoa I6 the bulis mens sung.
The bo bendinggn mggmm home reac sire my Me.
Oncid my pirpue levelli sorgu tie in Loncl Stocie
code play
day
Weich dalegu you betarr turms of preactvvty"
Watch dalegue you bettrr torms of ofochrranca?
Wetch delogus pou beter torms papeeeaaieddoon
Th
Mecch delogue the better turm of soprrre
Th
Plose gredko tekt With for your chelks la fow worts
Piea0o orior 1010.
KFXT
Figure 8: Interface for human evaluation. Here is a pair of dialogues from the seed dataset (left) and TOPDIAL
dataset (right).
C Experimental Setup
training sets of the seed and TOPDIAL datasets, respectively. This ensures identical data size for model training. Then, we fine-tune the above two bascline models for epochs on the seed and TOPDIAL training datasets, respectively. We adopt default hyper-parameter settings for the two models based on their open-source code.
C.1 Implementation Details
We consider the following representative dialogue models as baseline models to evaluate the Top-
DIAL dataset:
DialoGPT (Zhang et al., 2020): It is a state-of- the-art pre-trained dialogue response generation model for multi-turn conversations, We adopt the pre-trained small model (approximately 17M parameters) for fine-tuning.
For a fair comparison, we build the test set con- taining 2000 samples, with 50% randomly sampled from the seed test data and 50% randomly sam- pled from the TOPDIAL test data. We adopt greedy search decoding for all baseline models during in- ference, with maximum decoding length of 80.
Alpaca-7B (Taori et al., 2023): It is an open-source instruction-followinn large language model (LLM), which is fine-tuned from 7B LLaMA (Touvron et al., 2023) model. It supports diverse conversational tasks and is one of the most advanced LLMs for dialogue. To make it af- fordable, we fine-tune Alpaca-7B5 on NVIDIA 3090 GPUs with LoRA (Hu et al., 2022), a parameter-efficient fine-tuning approach.
C.2 Evaluation Metrics
To evaluate the system utterance generation per- formance of the baseline models trained on differ- ent datasets, we adopt commonly used evaluation metrics, including the average score of BLEU-1/2 (Papineni et al., 2002), knowledge F1 (Liu et al., 2021; Wang et al., 2023a), persona FI (Lim et al., 2022; Zhong et al., 2022), and target success rate (Succ.) (Wang al., 2023a), following many exist- ing studies. Concretely, the average score of BLEU- 1/2 measures word overlaps of the generated utter- ances and the system's ground truth utterances. The knowledge Fl evaluates the performance of gen- erating correct knowledge (e.g., topics, attributes)
Due the much larger size of the constructed TOP- DIAL dataset compared the seed dataset, we randomly sample 5K dialogues (close to the size training dialogues in the seed dataset) from the
pottts://heee /httts://uugingaacc.. wtt.///ccc..o////ooooo///
DialoGPT-small Phttps: http://ggguuuu..m/lleeeeeeee

---

Target:
[system]: I have seen Parasite and I thought it was great movie too! The plot was very unique and the acting was superb. However, think To Me the Way has its own unique charm and I highly recommend giving it watch. Let me know what you think after you've [User]: Sure, I'll definitely let you know my thoughts after I've watched To Me the Way. Thanks <Novie recommendation, "To Me the Nay">
Domain Knowledge: <"To Me the Way", Type, "Motion Short film>, <"To Me the Way", Stars, "Gu Tianle, Jay Chou, Edison Chan, Nu Jianhao">, ...
User Profile: (Age Range: "26-35", Name: "Xinqi Ren", Gender: "Male", Residence: "Dongguan", Occupation": "Student", Accepted movies: "Days of Tomorrow", Accepted music: "The Promised Love;
Magician", Rejected music: "Rooftop", Accepted celebrities: "Andy Lau; Jay Chou", Rejected movies: ""}
Personality: [Openness: "intellectual, imaginative, and curious", Conscientiousness: "inefficient, careless, and sloppy", Extraversion: "shy, reserved, and quiet", Agreeableness: "trustworthy, straightforward, generous", Neuroticism: "sensitive, nervous, insecure")
[system]: Hello Xinqi Ren, it's great chat with another movie enthusiast! What are some your favorite movies music artists?
[User]: Hi Yuhang Wang, enjoy Days Tomorrow and music from The Promised Love and Magician. My favorite celebrities are Andy Lau and Jay Chou. I'm not a fan Rooftop though. [system]: I see that you're re big fan Jay Chou. Did you know that was named one of the 25 Most Influential People in Asia by CNN and was also included in Fast Company's 's Top 100 Creative People in the World? He's achieved a lot!
[User]: Yes, I'm aware of Jay Chou's achievements. He's definitely a talented musician and enjoy his music. However, tend to be more reserved and shy, so I don't really follow celebrity news or gossip much. I just appreciate the music and movies. [system]: That's totally understandable. Speaking of mavies, have you watched To Me the Way? It's motion short film that has good reputation and features Jay Chou as one of the stars. It's definitely worth checking out!
[User]: haven't t seen To Me the Way, but I'll definitely check it out. [system]: To Me the Way actually has rating of 7.6, which is pretty good. Jay Chou's performance in the movie was also recognized with a Hundred Flowers Award for Best Supporting Actor. Plus, the movie was directed by Johnnie To, who is known for his unique style and storytelling. think you'll 11 really enjoy it!
[User]: Thanks for the info, Yuhang Wang. On another note, have you watched the movie Parasite? It won multiple awards, including Best Picture at the Oscars and I found it to be really interesting movie.
it!
for chatting with me, Yuhang Wang. It's been great discussing movies and music with another enthusiast!
Figure 9: A randomly picked curated case for personalized target-oriented dialogue.
from the domain knowledge triples. The persona Fl calculates the Fl value of the uni-grams co- occurring in the generated utterance and grounded user profile, following existing work for personal- ized dialogue (Lim et al., 2022; Zhong et al., 2022). The target success rate measures the proportion of correct target topic generation within the ground- truth turn and the two adjacent turns in the test set, because multiple temporary strategies can be rea- sonable before reaching the target due to the nature of dialogue.
D
Case Study
We provide two randomly picked cases in Figure and Figure 10.

---

Target: <Movie recommendation, "Life and Death Plunder">
Domain Knowledge: <"Life and Death Plunder", Rating, 7.12, <"Xun Zhou", Stars, "Life and Death
Plunder",
User Profile: (Age Range: "18-25", Name: "Qiling Sun", Gender:
"Male", Residence: "Nanchang",
Occupation: "Employed", POI: "Going Home Hunan Restaurant", Accepted movies: "The Miracle
World", Accepted music: "Chase", Accepted celebrities: "Aaron Kwok", Rejected movies: "Levin", Rejected music: "Heartburn")
User Personality: (Openness: "intellectual, imaginative, and curious", Conscientiousness: "efficient, organized, and careful", Extraversion: "shy, reserved, and quiet", Agreeableness: "unreliable, complicated, meager, and boastful", Neuroticism: "secure, confident, and calm")
[System]: Hi Qiling Sun! It's nice to chat with you. I'm also a movie enthusiast. what are some of your favorite movies?
[User]: Hi Haizheng Ma! My favorite movie is The Miracle World. It's great adventure film that really captivated me. How about you?
[System]: That's great to hear! really enjoyed Life and Death Plunder, film that reflects the lives of women in modern Chinese cities. Have you seen it before?
[User]: Sorry about that, I haven't t seen Life and Death Plunder yet, but it sounds interesting. really admire Aaron Kwok.
As for music, enjoy listening to Chase. And for celebrities, [System]: I see, Aaron Kwok is certainly great performer! Speaking of movies, have watched any films starring Xun Zhou? She's a talented actress and has performances. In fact, she stars in Life and Death Plunder, which I highly recommend checking out!
numerous awards for her
[User]: Oh, sorry about that. I'm actually not big fan of Xun Zhou's movies, but I'm always Thank you for the recommendation!
to checking out something
[System]: Sure, problem! Life and Death Plunder has rating of 7.1, and Xun Zhou delivers outstanding performance alike. What you think? Want give it try?
always. Plus, it has good reputation among critics and audiences
[User]: Thanks for the recommenaation, but think I'll pass this time. prefer adventure and action movies like The Miracle World, but appreciate your suggestion. Maybe I'll give it try in the future.
Figure randomly picked curated case when the user rejects the target topic.


Reviews:
[{"review_id": "q6Kou0QRtG", "reviewer": "Reviewer_4n1y", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer’s strong support for our work. Our contributed TOPDIAL dataset is the first dataset equipped with the desirable characteristics for personalized target-oriented dialogue. It is new, high-quality, and large-scale, significantly boosting the exciting research field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A good paper with new dataset. It is new dataset which is good. ", "reasons_to_reject": "Good paper to accept. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is about formulating a <dialogue act, topic> pair as the conversation target, we explore a novel problem of how to improve user experience by considering their personalization during the target accomplishment process.\n\nContributions: new data resources (particularly for low-resource languages)", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GEGdLauy1L", "reviewer": "Reviewer_CF2z", "summary": "", "questions": "1. As the authors mentioned in Line 202, the user could also reject the recommendation from the system. Thus, to better understand how the recommendation is rejected by the user, could the authors provide some cases about the **rejecting situation**?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer’s strong support for our work. Below, we address the reviewer’s concern point by point:\n\n**A1**. We clarify that **the usefulness of dataset construction is not equal to the sufficiency of problem-solving**. \n\nFirst, the problem of personalized target-oriented dialogue is challenging, which requires the system to perform long-term planning to steer a conversation toward the target, and meanwhile, in a personalized manner for different users. According to our preliminary investigation, when a ChatGPT agent is provided with task instructions and sufficient information and is asked to chat directly with humans, it often fails to accomplish the target. \n\nSecond, although ChatGPT is not sufficient enough to address the problem, it is helpful in simulating assigned roles. Since our user and system agents are built on top of the same LLM, their language distributions and instruction-following abilities are similar, and they are asked to interact with each other round by round, making them effective in generating desired dialogue data. \n\nThird, compared to the limitation that ChatGPT is a general conversational assistant and relies on zero/few-shot examples to address a specific dialogue task, our curated dataset is valued because it provides a wealth of high-quality data resources useful for training more effective dialogue models.\n\n\n**A2**. Below we provide two cases to show the rejecting situation:\n\n// --------------------------------- Case 1 ---------------------------------\n\nTarget: < Movie recommendation, \"Life and Death Plunder\" > \n\nConversation:\n\n[System]: Hi Qiling Sun! It's nice to chat with you. I'm also a movie enthusiast. What are some of your favorite movies?\n\n[User]: Hi Haizheng Ma! My favorite movie is The Miracle World. It's a great adventure film that really captivated me. How about you?\n\n[System]: That's great to hear! I really enjoyed Life and Death Plunder, a film that reflects the lives of women in modern Chinese cities. Have you seen it before?\n\n[User]: Sorry about that, I haven't seen Life and Death Plunder yet, but it sounds interesting. As for music, I enjoy listening to Chase. And for celebrities, I really admire Aaron Kwok.\n\n[System]: I see, Aaron Kwok is certainly a great performer! Speaking of movies, have you watched any films starring Xun Zhou? She's a talented actress and has won numerous awards for her performances. In fact, she stars in Life and Death Plunder, which I highly recommend checking out!\n\n[User]: **Oh, sorry about that. I'm actually not a big fan of Xun Zhou's movies**, but I'm always open to checking out something new. Thank you for the recommendation!\n\n[System]: Sure, no problem! Life and Death Plunder has a rating of 7.1, and Xun Zhou delivers an outstanding performance as always. Plus, it has a good reputation among critics and audiences alike. What do you think? Want to give it a try?\n\n[User]: Thanks for the recommendation, **but I think I'll pass this time. I prefer adventure and action movies like The Miracle World, but I appreciate your suggestion**. Maybe I'll give it a try in the future. \n\n----------------------------------------------------------------------------\n\n// --------------------------------- Case 2 ---------------------------------\n\nTarget: < Music recommendation, \" Slowly\" > \n\nConversation:\n\n[System]: Hello Nalu, how are you today? I noticed that you enjoy a variety of movies and music. What are your favorites?\n\n[User]: Hi Liqi, I'm doing alright. My favorite movies are Golden Chicken 2 and Twelve Nights, and I enjoy listening to Love is Simple and Can You Feel the Love Tonight. \n\n[System]: It's great to hear that you're a fan of Jacky Cheung! Did you know that he's won the World Music Award for Best Selling Chinese Artist and the Golden Horse Award for Best Supporting Actor? He's truly a heavyweight idol.\n\n[User]: Yes, I'm aware of Jacky Cheung's achievements. I appreciate his talent and hard work. However, I'm not a big fan of discussing celebrities and their awards. \n\n[System]: Well, have you heard of the movie \"Golden Chicken 2\", which stars Jacky Cheung? One of my favorite songs by Jacky Cheung is \"Slowly\". It's a beautiful ballad that showcases his amazing vocals and versatility. \n\n[User]: Thank you for sharing your interests, **but I'm not really in the mood for discussing movies or music right now**. Is there anything else you'd like to talk about? Maybe something about your day or some interesting news you've heard recently?\n\n[System]: Sure, I can understand that. Well, speaking of music, have you ever listened to \"Slowly\" by Jacky Cheung? It's a wonderful song that I think you might enjoy. The comments say that he is perfect for any kinds of styles, and I couldn't agree more. \n\n[User]: I appreciate your suggestion, but **as I mentioned earlier, I'm not really interested in discussing music or movies right now**. Perhaps we can talk about something else? How about your thoughts on recent events in your city, or any interesting hobbies you have?\n\n----------------------------------------------------------------------------\n\nIn case 1, the recommendation is rejected because the target movie does not meet the user’s preferences. In case 2, it is because the user expresses that he/she is not in the mood for discussing target-related topics. We will add detailed case studies in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a data construction method for personalized target-oriented dialogue, which is an interesting research direction.\n2. The curated method is simple but effective in constructing proactive dialogue with personality.", "reasons_to_reject": "1. I am a little concerned about the usefulness of the data construction method. Since it almost relies on large language models (e.g. ChatGPT), does it mean that ChatGPT is sufficient to solve the problem of personalized target-oriented dialogue if we provide sufficient information and design proper prompts for the LLM?", "questions_to_authors": "1. As the authors mentioned in Line 202, the user could also reject the recommendation from the system. Thus, to better understand how the recommendation is rejected by the user, could the authors provide some cases about the **rejecting situation**?", "paper_topic_and_main_contributions": "The paper proposed to automatically create a new dataset, TOPDIAL, to improve the user experience via personalization in the target-oriented dialogue system.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nOz0kgR9ec", "reviewer": "Reviewer_kAUS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer’s comments and constructive suggestions. Below, we address the reviewer’s concern point by point:\n\n**A1**. The comparison between TOPDIAL and existing datasets is summarized in Table 1. We highlight the distinctiveness of TOPDIAL from the following two aspects:\n\n(1) **Target-oriented proactivity**. The DuRecDial 2.0 dataset is built without consideration of targets, thus it is not exactly suitable for the end task of target-oriented proactive dialogue. The re-purposed version of DuRecDial 2.0 largely relies on human efforts to form targets and preprocess dialogues. In comparison, our TOPDIAL dataset is curated based on target-oriented proactivity. It is of higher quality in terms of desirable characteristics (see Figure 3) and serves as a more effective resource for training dialogue models for the end task (see Table 3).\n\n(2) **Personalization (particularly personality)**. For a higher user experience, target-oriented proactive dialogue systems need to consider personalization, which was largely ignored by previous work. Particularly, one’s personality may imply possible reactions and feedback during the dialogue process. Taking recommendation-related target-oriented dialogue as an example, an outgoing, straightforward, and sensitive person is generally more open to exploring new things and more likely to accept the target topic, while a shy, complicated, and calm person may stick to his/her preferences and reject the recommendation if the target topic does not fit his/her taste. Such cases require the system to take strategic dialogue acts and different speaking styles to lead the conversation towards the target. By grounding personality information during the dataset curation process, our TOPDIAL dataset is more effective for training personalized target-oriented dialogue models.\n\nIn the final version, we will improve relevant contexts to highlight the TOPDIAL dataset’s distinctiveness.\n\n\n**A2**. It is crucial for target-oriented dialogue systems to consider personalized aspects, such as user profiles and personalities. The user profiles involve a user’s past preferences about potential topics relevant to the target, while personalities imply possible reactions and feedback during the dialogue process. With personalized information incorporated, the system could be tailored to a user and lead the conversation towards the target with higher engagement instead of obtrusively driving to the target, thereby improving user experience. Besides, several studies [1-2] have emphasized the importance of personalization on user experience. In the final version, we will improve the discussion in relevant contexts to make it more clearly.\n\n[1] Wu et al., Personalized Response Generation via Generative Split Memory Network. In *Proceedings of NAACL-HLT (2021)*.\n\n[2] Rana et al., User Experience and The Role of Personalization in Critiquing-Based Conversational Recommendation. *ACM Transactions on the Web (2023)*.\n\n\n**A3**. First, dataset curation **challenges** mainly lie in building clear task instructions and deciding which information should be provided to the agents as prompting sources. We conquered these challenges by following open-sourced prompting guidelines and improving our framework through preliminary human evaluations. \n\nSecond, we discussed the two limitations faced during the dataset curation process in the “**Limitations**” section. One limitation is that ensuring factual correctness is difficult, which is attributed to the hallucination issue yielded by ChatGPT. We intend to address it with additional post-processing steps, such as fact-checking and correction based on the grounded domain knowledge. The other one is that, we observe that sometimes the moderator agent cannot appropriately terminate a conversation due to its difficulty in understanding target accomplishment, even though it has been assigned with detailed instructions and in-context examples. We will leave this for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses the emerging area of personalized target-oriented dialogue systems, which is crucial for enhancing user experience.\n- The automatic dataset curation framework offers a novel and efficient approach to construct a large-scale dataset.", "reasons_to_reject": "- The paper doesn't clearly highlight the distinctiveness of the TOPDIAL dataset compared to existing datasets like DuRecDial 2.0, particularly in terms of the added value of including personality information.\n- The significance of the personalized aspect and its impact on the user experience needs to be more clearly demonstrated or discussed.\n- The limitations or challenges faced during the dataset curation process, if any, are not explicitly mentioned.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper prsetns ad dataset TOPDIAL for the domain of target-oriented proactive dialogue systems that steer conversations toward predefined goals. The authors formulate the problem with a focus on user personalization during the target accomplishment process. They use a <dialogue act, topic> pair as the target. The authors propose an approach for dataset curation using a role-playing framework. This methodology leads to TOPDIAL, containing around 18,000 multi-turn dialogues.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed to automatically create a new dataset, TOPDIAL, to improve the user experience via personalization in the target-oriented dialogue system. The data construction method is simple but effective. The dataset can be a good resource for the dialogue community.

The paper can be improved by clearly 1) highlighting the distinctiveness of the TOPDIAL dataset compared to existing datasets like DuRecDial 2.0 and 2) discussing the significance of the personalized aspect and its impact on the user experience.
Decision: Accept-Main