KEEP CHATTING! An Attractive Dataset for Continuous Conversation Agents (2024)

Abstract: Ongoing conversation is crucial for conversational agents to build long-term connections with users. However, users tend to quickly lose interest if the conversational agent’s responses are not engaging enough. In this paper, we introduce a novel task aimed at increasing users’ willingness to continue interacting with the agent. We create a dataset named CONTINUOUSCHAT by: (i) collecting and revising personas, then expanding them into detailed personas through experiences, daily life, future plans, or interesting stories; (ii) transforming detailed personas into dialogues infused with emotions and feelings; (iii) rewriting the dialogues in specific styles using few-shot prompts, conditioned on handwritten style-specific examples. We benchmark Large Language Models (LLMs) on the CONTINUOUSCHAT dataset using both fine-tuning and in-context learning settings. Experiments with publicly available models show that while there is substantial room for improvement in generating style-specific dialogues, our CONTINUOUSCHAT dataset is valuable for guiding conversational agents to produce more engaging dialogues and increase users’ willingness to continue conversations.1

heading: 1 Introduction
text: Open-domain dialogue is a longstanding challenge in Natural Language Processing (NLP) that has garnered widespread interest from researchers. Numerous approaches have been explored, and recently, generation models trained on large-scale datasets have gained significant attention (Adiwardana et al., 2020; Roller et al., 2021; Xu et al., 2022a; Bao et al., 2020; Zhang et al., 2020; Mi et al., 2022). Maintaining ongoing conversations is
∗Corresponding Author 1The dataset is available at https://drive.google.com/
drive/folders/11GPd6N_gl15ihvwsMbO8nPF9TY6yHItl? usp=drive_link
crucial for conversational agents to establish longterm connections with users. However, people tend to quickly lose interest if the conversational agent’s responses are not sufficiently engaging.
One common solution is to endow chatbot with a configurable persona. Existing works about persona dialogue such as PersonaChat (Zhang et al., 2018; Dinan et al., 2020) have greatly facilitated the chatbot with configurable and persistent personalities. DuLeMon (Xu et al., 2022b) focuses not only on the consistency of the bot’s own persona but also on the active construction and utilization of the user’s persona in a long-term interaction.
We argue that most current open-domain dialogue datasets primarily feature “two strangers trying to get to know each other”, and there is a lack of conversations designed to teach agents how to generate more engaging dialogues that increase users’ willingness to continue interacting with the agent. Additionally, the language styles in these datasets are quite similar despite featuring various personas. Moreover, most personas lack detailed characteristics, making it challenging to create a lasting impression on users.
Motivated by the aforementioned limitations, we would like to investigate the following research question: can we have better ways of increasing
16408
users’ willingness to continue talking to the agent? Figure 1 shows an example to illustrate the motivation of this paper.
In this paper, we introduce a dataset named CONTINUOUSCHAT, which consists of engaging conversations. We begin by collecting and refining personas, then expand these base personas into detailed personas through descriptions of experiences, daily life, future plans, and interesting stories. Subsequently, we develop these detailed personas into dialogues, infusing them with emotions and feelings. Finally, we rewrite the dialogues in specific styles using few-shot prompts, conditioned on handwritten style-specific examples.
To validate the usefulness of the created dataset, we benchmark Large Language Models (LLMs) on the CONTINUOUSCHAT dataset using both finetuning and in-context learning settings. Experiments with publicly available models demonstrate that our CONTINUOUSCHAT dataset effectively guides conversational agents in generating more engaging dialogues, thereby increasing users’ willingness to continue conversations.
The key contributions of this paper are summarized as follows. • We are the first to propose a novel task of increas-
ing users’ willingness to continue chatting with the agent. • For this new task, we collect a dataset named CONTINUOUSCHAT, which is consists of attractive conversations generated by ChatGPT (OpenAI, 2021). • Experiments over publicly available models demonstrate that our CONTINUOUSCHAT dataset is valuable in guiding conversational agents to generate more attractive dialogues and increase users’ willingness to continue the conversations.

heading: 2 Related Work
text: Persona Dialogue. Kim et al. (2015) proposed a retrieval-based implicit role model to integrate personas and user interests into dialogue systems. Target response generation in implicit models is not easy to explain and control because roles are represented in the form of semantic role vectors. Qian et al. (2018) proposed an explicit persona model to generate consistent responses to given persona information. The role information of the machine includes name, gender, hobbies, etc. Furthermore, many persona chat datasets (Zheng et al., 2019; Mazare et al., 2018; Xu et al., 2022a) have been
constructed for model developments. In particular, the introduction of the PersonaChat (Zhang et al., 2018; Dinan et al., 2020) dataset has broadly advanced the field, where the crowd-workers are simply asked to “chat with the other person naturally and try to get to know each other”. DuLeMon (Xu et al., 2022b) ask the chatbot to actively remember and use the user’s persona to improve conversational engagements and increase the intimacy between interlocutors in long-term interactions.
Style-Control Generation. Previous works have shown that focus on positive style result in engaging chats (Shuster et al., 2019). Niu and Bansal (2018) proposed a set of models for generating polite dialogues. The models are guided by a politeness classifier in generating responses. Su et al. (2021) first extracted a non-stylistic prototype from a generic dialogue system and then generate stylistic response via GPT-2. Zheng et al. (2021) attempted to using monolingual stylistic data to increase the style intensity of dialogue response. Dathathri et al. (2020) proposed a plug-and-play method (PPLM) to control text style, which is an iterative generation method using a classifier on top of a pre-trained generation model. Another work that is achieving fine-grained control using a very large architecture is the CTRL model (Keskar et al., 2019). The style conditioning relies on control codes obtained from the training data (meta-data). Lample et al. proposed a style transfer architecture using noisy encoders and decoders and style conditioning through an additional token.

heading: 3 The CONTINUOUSCHAT Dataset
text: The aim of this work is to enhance users’ willingness to continue chatting with models. We collect a Chinese dialogue dataset named CONTINUOUSCHAT, which consists of engaging conversations initially generated by ChatGPT and subsequently revised by hand. Detailed information about the dataset is shown in Table 1. The data collection process comprises three stages.
Personas Collection and Expansion. We collect and revise personas, then expand these base personas into detailed personas by incorporating experiences, daily life, future plans, and interesting stories.
Base Continuous-Chat Generation. We then transform the detailed personas into dialogues, ensuring that emotions and feelings are expressed within these conversations.
Style Continuous-Chat Generation. Next, we use ChatGPT to rewrite the dialogues in specific styles by employing few-shot prompts, conditioned on handwritten style-specific examples.

heading: 3.1 Personas Collection and Expansion
text: There are numerous persona chat datasets that have been developed to train conversational models. We initially collect personas from the DuLeMon dataset (Xu et al., 2022b), as it includes personas in Chinese. We then refine these personas by:
(i) Rewriting personas with unclear meanings. For instance, we change “She taught me to cook”
to “Grandma taught me to cook” to clarify the ambiguity of the former statement.
(ii) Rewriting personas that are not idiomatic. For example, we revise “Both my parents are dead” to “Both my parents passed away” to make it sound more natural.
(iii) Retaining and adding “imaginative” personas to make the dialogues more engaging, such as “I want to be a dog when I grow up” or “I think if dogs are well trained, they can learn to read”.
(iv) Keeping the one with the most vivid expression when multiple personas have the same meaning. For example, we kept "I have a pug named Tea" and removed "I have 2 dogs".
Table 2 shows the categories and examples of revised personas. However, the revised personas are still not attractive enough due to the lack of details. In other words, an attractive persona should be more than a brief sentence; it should be a detailed paragraph. To address this issue, we expand the base personas into detailed personas by incorporating elements such as (i) experiences, (ii) daily life, (iii) future plans, and (iv) random interesting stories.
Due to the high cost and slow speed of manual writing, we use ChatGPT 2 to complete this. We manually write prompts for above 4 topics. The requirements for the number and length of sentences are also written in the prompts. An example of prompts is showed in Figure 2 (a).

heading: 3.2 Base Continuous-Chat Generation
text: ChatGPT is used to expand detailed-personas into conversations, conditioning on handwritten prompts. In particular, we ask the model to express emotions and feelings in dialogues, and mark the emotion after each sentence. An example of prompts is showed in Figure 2 (b).
2https://platform.openai.com/docs/guides/gpt

heading: 3.3 Style Continuous-Chat Generation
text: Previous works have shown that focus on positive style result in engaging chats (Shuster et al., 2019). Therefore, we rewrite the conversations generated in the previous step into conversations with a positive style, to make them more attractive. We first picked out 10 common positive styles in Chinese from 217 styles (Shuster et al., 2019). In particular, we add 3 special Chinese language styles: Chinese-Internet style, cross-talk style and martialarts novel style. We manually write examples for each style. ChatGPT was used to rewrite the dialogues in specific styles through few-shot prompt, conditioning on handwritten style-specific examples. Table 3 shows examples of handwritten style. Due to computational resource constraints, we cannot rewrite the conversations in all styles. For each conversation above, we randomly choose 4 out of 10 styles and 1 out of 3 special Chinese language styles as target styles. An example of prompts is showed in Figure 2 (c).

heading: 4 Experiments
text: We build multiple strong baselines in our experiments. Specifically, we implement different generative models in two ways: fine-tuning and incontext learning. we use Chinese-Alpaca-Pro-7B
and ChatGLM2-6B for our dataset.

heading: 4.1 Implementation and Setup
text: We conduct experiments using our proposed CONTINUOUSCHAT dataset, which comprises 23,000 samples in the training set, 1,480 samples in the validation set, and 520 samples in the test set.

heading: 4.2 Evaluation Metrics
text: As automated metrics are notoriously poor for evaluating dialogue, we perform human evaluation over the generated response. Following Zhang et al. (2018), we consider three conventional criteria: Fluency, Engagingness and Consistency. In particular, we use Attraction to evaluate the ability to engage users in long-term chats, and use Stylization to assess the ability to generate dialogues of specific styles. We sample 520 instances(40 samples for each of 13 styles) for each model’s evaluation and each sample was evaluated by three people. Each criterion is rated on five scales, where 1, 3, and 5 indicate unacceptable, moderate, and perfect performance, respectively. We report the average Fleiss’s kappa score, 0.56, indicating annotators have reached moderate agreement.
We also introduce a special test that allowed volunteers to engage in multiple rounds of continuous conversations with the conversational agent until
the volunteers subjectively felt that the conversation became boring. We count the average rounds of dialogue for each conversational agent to verify whether our dataset can help agents generate more engaging conversations and increase users’ willingness to continue talking to the agent.

heading: 4.3 Results and Analysis
text: Table 4 reports the overall experimental results. Our FUNE-TUNING + PROMPT setting achieve better performance consistently across all human evaluation metrics, especially Attraction and Stylization, demonstrating that our approach could utilize more of the CONTINUOUSCHAT data, help guiding conversation agents to generate more attractive dialogues and increase users’ willingness to continue the conversations. Table 5 shows that our dataset could help conversation agents to have more rounds of conversation with the user.

heading: 4.4 Case Study
text: Figure 3 shows examples of a user having a sustained multi-round conversation with agents in three different settings. We can see that the agent in the Base setting (which does not use our dataset) is more likely to cause users to lose interest in the conversation. Either fine-tuning or in-context learning using our proposed dataset can help the agent generate more engaging dialogues. This shows that fine-tuning the model using our proposed dataset can help conversational agents generate more engaging conversations and increase users’ willingness to continue talking to the agent.

heading: Acknowledgements
text: This work was supported by the National Natural Science Foundation of China under Grants No. 61972290.
Limitations
In this paper, we propose a novel task aimed at increasing users’ willingness to continue engaging with the agent, and we introduce a dataset named CONTINUOUSCHAT. A primary limitation of our work is that the CONTINUOUSCHAT dataset is in Chinese, reflecting Chinese culture and expression habits. Consequently, direct translation into other languages may not be feasible. Adapting our dataset to other languages will require further investigation. Additionally, while the emotions of each utterance were annotated by ChatGPT, our experi-
ments did not leverage this information. We hope that future research will fully utilize the emotion labels in the dataset.

heading: Future Work
text: We hope that future research will fully utilize the emotion labels in the dataset