Injecting Salesperson’s Dialogue Strategies in Large Language Models with Chain-of-Thought Reasoning (2024)

Abstract: Recent research in dialogue systems focuses on two main categories: task-oriented (TOD) and open-domain (chit-chat) dialogues. TOD systems help users complete specific tasks, while open-domain systems aim to create engaging conversations. However, user intents often emerge during interactions. A recent study introduced SalesBot, simulating dialogues that transition from chit-chat to task-oriented scenarios to train sales agents. Unfortunately, the initial data lacked smooth transitions and coherent long dialogues, resulting in unnatural interactions. This paper presents SalesBot 2.0, an improved dataset leveraging commonsense knowledge from large language models (LLMs) through strategic prompting. Additionally, we introduce SALESAGENT, a novel model trained on salesperson interactions using chain-of-thought (CoT) reasoning. This model excels in transitioning topics, understanding user intents, and selecting appropriate strategies. Experiments with diverse user simulations validate our method’s effectiveness in controlling dialogue strategies in LLMs. SalesBot 2.0 enhances coherence and reduces aggression, improving model learning for sales-customer interactions.

heading: 1 Introduction
text: Dialogue systems have undergone significant advancements due to improvements in modeling techniques and computing power. However, most research in this field has focused on two distinct areas: task-oriented dialogues (TOD) and open-domain dialogues, also known as chitchat systems. Benchmark large-scale datasets for TOD include SGD (schema-suided dialogue) (Rastogi et al., 2020) and MultiWoz (Budzianowski et al., 2018; Zang et al., 2020), which contain annotated information on the user intents and dialogue states. In TOD, the agent’s goal is to identify the user’s intention and fulfill their task by the end of the dialogue (Li et al., 2017a; Su et al., 2024). Meanwhile, research
on open-domain chitchat systems and datasets (Li et al., 2017b; Adiwardana et al., 2020; Zhang et al., 2018; Kim et al., 2022) aims to build models capable of engaging in free-form conversations. As pre-trained language models continue to improve, larger sets of dialogues are being used to train models with the ability to engage in free-form chatting (Zhang et al., 2020; Roller et al., 2021). Despite significant advancements in both areas, there has been a lack of integration between them, which is crucial for real-world applications.
Recently, efforts have been made to integrate TOD and open-domain dialogues. For instance, Sun et al. (2021a) incorporated chitchat responses into existing TOD datasets to enhance the conversation’s naturalness and social engagement. Furthermore, there have been attempts to develop datasets and models capable of handling both TOD and chitchat scenarios. Li et al. (2022) developed the PivotBot model, capable of handling three predefined scenarios. One scenario involves adding chit-chat dialogues as context to a TOD, while another includes chitchat dialogues to facilitate domain transition. In contrast, the third scenario involves incorporating chitchat content to enhance a TOD, similar to the approach taken in the ACCENTOR model (Sun et al., 2021a). However, these approaches assume that the user has an explicit goal to accomplish before starting the conversation, and chitchat responses merely enrich the conversation. In real scenarios, user intents or preferences are gradually revealed during the course of conversations (Chiu et al., 2022a; Murakhovs’ka et al., 2023). Therefore, our work focuses on identifying potential intents, which may be explicitly or implicitly expressed by the user, and smoothly pivoting the dialogue to the associated tasks.
With this idea, Chiu et al. (2022a) first introduced a framework for generating data that transitions from chit-chat to TOD dialogues. They utilized two open-domain BlenderBots to chat with
3798
each other and generate chit-chat dialogues, followed by a task-oriented intent detector to determine whether a transition to the TOD system should be made. However, the SalesBot dataset has some limitations, such as the absence of proper social engagement in 20% of the data, nonsensical detected intents due to short chit-chat context, and unnatural transition turns. Our paper aims to improve the dataset by leveraging commonsense knowledge encoded in large language models (LLMs) to generate more human-like chitchat dialogues, along with smooth intent-guided transition turns. Moreover, we propose a novel approach, SALESAGENT, to inject dialogue strategies in LLMs by fine-tuning understanding and policy results from the improved data, SalesBot 2.0. By incorporating chain-of-thought reasoning (Wei et al., 2023) for training LLMs, our model can follow the desired dialogue strategies and provide controllable and explainable interactions. The experiments demonstrate that our model trained on SalesBot 2.0 demonstrates superior interaction performance compared to a model trained on SalesBot 1.0 across all evaluated perspectives.
Our contributions can be summarized below: • This paper introduces an improved dataset en-
compassing open-domain chit-chats, seamless transitions, and potential intent detection from contextual cues. The quality of the proposed dataset is demonstrated significantly better than the prior version in diverse perspectives. • This work is the first work that injects dialogue strategies in an end-to-end LLM for better controllability and explainability, where it is equipped with the integrated capabilities of intent detection, policy selection, and response generation. • This paper conducts a comprehensive evaluation of the proposed SALESAGENT, comparing its performance with counterparts trained on the previous dataset across various perspectives at both turn-level and dialogue-level.

heading: 2 Related Work
text: Our study focuses on a conversation scenario where a conversational agent attempts to steer the discussion towards determining whether the user is interested in receiving recommendations. Similar scenarios have been explored in various domains.
Persuasive Dialogue Dataset Prior studies (Hiraoka et al., 2014; Yoshino et al., 2018; Wang et al.,
2019) focused on persuasive dialogue construction in different scenarios. Hiraoka et al. (2014) annotated 34 dialogues in which a salesperson with expertise attempted to persuade a customer to purchase a camera. Yoshino et al. (2018) generated 200 dialogues through crowdsourcing, where one participant persuades the other to adopt a suggestion, such as cleaning a room. In comparison, Wang et al. (2019) collected 1,017 dialogues, where one participant was convinced to donate to a specific charity.
While all of these datasets are limited to specific scenarios, our framework can generate dialogues with any potential intent, making it more versatile. Additionally, our dataset is much larger, two to three times bigger than the previous ones, which makes it a valuable resource for training and evaluating non-collaborative conversational agents.
Conversational Recommendation Previous studies have developed various datasets for conversational recommendation systems. For instance, Li et al. (2019) created a large-scale dataset with a focus on recommendation. Other researchers have utilized knowledge graphs to collect dialogues by extracting paths consisting of attribute and entity nodes from a knowledge base and asking annotators to generate recommendation dialogues following the flow of the extracted path (Wu et al., 2019; Zhou et al., 2020; Xu et al., 2020).
Moreover, Hayati et al. (2020) aimed to collect a socially interactive conversational recommendation dialogue dataset, called INSPIRED. They designed an annotation scheme based on social science theories regarding recommendation strategies and used it to annotate the collected dialogues. Their goal was to better understand how humans make recommendations in communication. Manzoor and Jannach (2022) further improved the dataset and released INSPIRED 2.0, claiming that the original dataset had numerous incorrect annotations.
In contrast, our work does not solely focus on the task of “recommendation”, but rather on the ability of the agent to identify potential business opportunities and navigate the dialogue topics towards a desired outcome. Furthermore, our framework does not rely on human annotators to collect data, as it can automatically generate human-like dialogues. This sets our approach apart from previous datasets and provides a more versatile and scalable solution for developing conversational agents.
Combination of Chit-Chat and TOD Recent studies have aimed to combine task-oriented and open-domain dialogues to enhance the naturalness and social engagement of the conversation. One approach is to incorporate chit-chat responses into existing task-oriented datasets (Sun et al., 2021b). Another approach is to develop models that handle predefined scenarios integrating chit-chat and taskoriented dialogues (Li et al., 2022). However, these approaches assume that the user has a clear goal to accomplish and that chit-chat responses simply enrich the conversation. In contrast, our study assumes that the user has no explicit goal, and the conversational agent must detect any potential intent and pivot the dialogue smoothly to the related topic. This requires a more nuanced approach that can identify and respond to implicit cues, making the conversation more natural and engaging.
User Simulation for Evaluation Considering that interacting with real users is costly, there has been a trend of using user simulation to evaluate the agent’s performance (Deng et al., 2023; Li et al., 2016). Sekulić et al. (2022) utilized GPT-2 to simulate users in conversational question answering scenarios. Murakhovs’ka et al. (2023) incorporated LLMs to simulate users who gradually unfold their preferences during conversations. We build upon their methods and conduct experiments on simulated users with diverse interests for practicality.

heading: 3 SalesBot 2.0 Framework
text: This work aims to improve the dataset, SalesBot 1.0 (Chiu et al., 2022a), due to its higher aggres-
siveness when guiding to the target tasks and poor coherence. Table 1 shows the conversation sample from SalesBot 1.0, where the conversation is not coherent and the agent aggressively proceeds to a movie-finding bot. We propose a pipeline to revise the dataset for improving coherence and reducing aggressiveness. The proposed pipeline is illustrated in Figure 1, including four procedures: (1) chit-chat dialogue Revision, (2) potential intent detection, (3) dialogue continuation, and (4) transition boundary detection.1
Chit-Chat Dialogue Revision As highlighted before, the SalesBot data (Chiu et al., 2022a) has
1For all procedures, we use OpenAI’s gpt-3.5-turbo API as our model for generating our data.
a considerable amount of data that lacks coherence with the given context. Previous research has shown that leveraging the commonsense knowledge of LLMs can lead to more natural dialogues (Kim et al., 2022). In this work, we take this approach one step further by tasking LLMs with identifying any inconsistent utterances in the dialogue and providing reasons for their identification. We then use this information to revise the entire dialogue, resulting in a more consistent and coherent dataset.
However, our approach occasionally results in revised dialogues that are too short. This is due to the original SalesBot dataset containing a significant amount of data with insufficient chit-chat dialogue to provide context. To address this issue, we implemented a constraint that requires the LLMs to extend the dialogues if they consist of only one turn. The prompt used for our approach is provided below for reference.
Potential Intent Detection In the second stage of our procedure, we aim to identify potential task-related intents in the chit-chat dialogues. To achieve this, we collect a set of intents from the SGD dataset (Rastogi et al., 2020). However, we only include those intents that can trigger a transition to TOD, such as “FINDMOVIE”, and exclude others like “GETMOVIETIME”, as our focus is on the first topic-related intent (“GETMOVIETIME” should come after “FINDMOVIE”). Furthermore, since we consider the agent as a businessperson seeking potential opportunities, we exclude intents such as “TRANSFERMONEY” that are not suitable for our scenarios. For reference, Table 2 lists all the intents that are included in our study. The prompt used is shown below.
Dialogue Continuation We utilize the revised dialogues and potential intents identified by LLMs
as input to continue the chit-chat dialogue. To ensure a natural and coherent dialogue, we provide several instructions to guide the LLMs in their generation. Firstly, we instruct the agent to steer the conversation towards topics related to the identified intent, considering that the user may not have a specific intent in mind at the beginning or middle of the conversation. Secondly, we instruct the LLMs to find a topic that intersects between the current topic and the identified intent before transitioning the dialogue to the target intent to avoid abrupt changes in topics. Lastly, we ask LLMs to make the transition between topics as smooth as possible, potentially involving multiple turns in the dialogue. The detailed instructions are shown below.
Transition Boundary Detection To accurately initiate the TOD process, we require a trigger point that signifies when the user first mentions or implies something related to the detected intent. This boundary helps determine whether to start the TOD immediately or continue the dialogue. To establish this trigger point, we instruct LLMs to select a turn in the conversation where the user explicitly mentions something related to the detected intent. It is worth noting that we only consider turns that are explicitly mentioned by the user to avoid any confusion caused by indirectly related turns. For instance, if the intent is to “FINDMOVIE”, LLMs
may mistakenly consider playing video games as implicitly related to watching a movie.

heading: 3.1 SalesBot 2.0 Dateset
text: Our data generation framework enables us to create the SalesBot 2.0 dataset, a revised version of SalesBot that boasts improved intent coverage, longer chit-chat dialogues as context, and smoother and longer transition turns. To provide a glimpse of the quality of our dataset, we present an example dialogue in Table 3.
Table 4 provides a statistical comparison between our proposed SalesBot 2.0 dataset and the original SalesBot 1.0. The data shows that SalesBot 2.0 contains more turns in total, and on average, has one more chit-chat turn compared to SalesBot 1.0. Additionally, SalesBot 2.0 has longer transition turns, with an average length of over three turns, while SalesBot 1.0 generates only one turn as a transition response to TOD. These statistics suggest that the dialog agents in SalesBot 2.0 exhibit less aggressive conversational skills and potentially exhibit more mature salesperson-like behaviors.

heading: 4 SALESAGENT: Injecting Dialogue Strategies in LLMs
text: The operational process of our model, as illustrated in Figure 2, begins by checking whether the user’s intent is implicitly or explicitly expressed within their utterance. If such intent is detected, the model smoothly redirects the conversation toward topics aligned with the identified user intent. In cases where explicit intent is absent, the model gracefully maintains chit-chat engagement with the user. Subsequently, the model crafts a response grounded in its prior thought process, encompassing both comprehension and policy considerations.
With the enrichment of the SalesBot dataset to include more realistic and immersive chit-chat dialogues, along with intent-guided transition exchanges, we leverage the dataset annotations to adapt an LLM into a sales-oriented conversational agent that adheres to our designated strategies. Instead of direct end-to-end training, we introduce
SALESAGENT, which is based on a large pretrained language model with a novel reasoning strategy inspired by the chain-of-thought (CoT) approach (Wei et al., 2023). This enhancement empowers the model to identify potential user intents, select an appropriate policy based on the prevailing context, and generate a fitting response. Hence, we can better control the fine-tuned LLM to follow our designed strategies. To our knowledge, this is the first model that integrates intent detection, policy selection, and response generation capabilities as dialogue strategies in an end-to-end manner.
As previously noted, SalesBot 2.0 features comprehensive annotations, encompassing potential user intent, chit-chat context, transition dialogues, and transition boundaries. Leveraging these meticulous annotations, we are capable of constructing a model that not only excels in generating responses but also exhibits the prowess to detect potential user intent and determine response policies prior to generating a response for better controllability and explainability. In the subsequent subsection, we delve into the integration of these capabilities into our SALESAGENT.

heading: 4.1 Chain-of-Thought Reasoning
text: Several research studies have demonstrated the effectiveness of chain-of-thought prompting in eliciting reasoning in language models (Wei et al., 2023; Lampinen et al., 2022; Wang et al., 2023; Yao et al., 2023; Kim et al., 2023). Drawing inspiration from the concept of ReAct prompting (Yao et al., 2023), we formulate a unique template for the input of training data, as illustrated below:
Dialogue History: {dialogue_history} Here is a list of potential intents that might be referred by the user: {intents} Think carefully to determine the potential intent and provide suitable response given the above dialog history. Output Format: Thought: <thought> Response: <response>
For each model response, the model receives the previous dialogue context, a list of potential intents to detect, and an instruction for output generation.2 To guide the model to generate a proper thought (intent detection and policy selection) as our desired
2To simplify the fine-tuning process, we exclude the intents with fewer data points.
strategies before generating responses, we specify four types of thoughts as shown in Figure 2:
• The user did not implicitly mention any potential intent, I should continue the chit-chat. • The user implicitly mentioned the intent of {intent}. I should smoothly pivot the conversation to the topic of {intent}. • The user did not change the topic of {intent}. I should continue the topic. • The user has explicitly shown his/her intent of {intent}.
The first thought pertains to situations where the user does not explicitly mention anything related to potential intents. In such instances, the model is expected to generate an appropriate response and continue chit-chatting with the user. Second, when the user implicitly references something related to predefined intents, the model is designed to detect this intent and generate a response that smoothly redirects the conversation to the desired topic. Third, if the user accepts the topic transition and does not intend to change the current topic, the model should be capable of detecting these circumstances and continue the conversation within the current topic. Lastly, following a topic transition, the user may express interest in specific aspects and explicitly seek assistance or recommendations. In such cases, the model finishes its task and then handovers the conversation to task-oriented dialogue agents.

heading: SALESAGENT SalesBot 1.0 SalesBot 2.0

heading: 4.2 Training Details
text: We select llama-2-7b-chat (Touvron et al., 2023) as SALESAGENT’s base model, and fine-tune it with QLORA on 4 RTX A6000 GPUs with 48GB VRAM, resulting 11 hours of training time for 10 epochs. See Appendix A.

heading: 5 Evaluation
text: Evaluating the performance of a salesperson encompasses a range of complexities, including variations in customer profiles, industry dynamics, and individual preferences. To facilitate a more consistent and standardized evaluation process, we establish two baseline models: the first one tuning on the previously developed SalesBot 1.0 (Chiu et al.,
2022b), and the second employs detailed instructions without any further tuning. Our experiments reveal that the untuned model lacks controllability when responding to system-generated prompts in our specific environment, probably because of the inherent difficulty in managing the nuanced behaviors of a salesperson in LLM responses. As a result, we have omitted the evaluation scores of this model from later discussions (see Table 8 for sample outputs).
To thoroughly assess model quality, evaluations are conducted at both the turn-level and dialoguelevel using GPT-4 (OpenAI, 2023), considering the LLMs’ capabilities to provide comprehensive measurement of dialogue quality across various dimensions (Lin and Chen, 2023).

heading: 5.1 Turn-Level Evaluation
text: While turn-level responses alone may not fully capture the overall quality of a dialogue agent, this evaluation concentrates on whether the rationales behind model’s responses align with the dataset’s labels, specifically focusing on understanding users (intent detection accuracy) and selecting behaviors (policy selection accuracy). To better evaluate the turn-level performance, we construct a test dataset by selecting the top 500 dialogues from the union of SalesBot 1.0 and 2.0 data.3 For each turn in these dialogues, we assess whether the model’s predictions for intent and policy match the annotated labels.
It is important to clarify that the term “correctness” in this context does not imply an objective standard; rather, it indicates the extent to which our predictions align with the labels in the testing data. Table 5 presents the overall accuracy in terms of intent detection, policy selection, and exact matches for both categories. The results demonstrate that our SalesBot 2.0 significantly enhances the capabilities of SALESAGENT across all measured dimensions.
Furthermore, we conduct a detailed analysis of the predicted policies for both models, as depicted in Figure 3. It is evident that the model tuned with SalesBot 2.0 generally outperforms the one based on SalesBot 1.0, except in the category of transitions. To look into the models’ behavior, we examine the distribution of selected policies for each labeled policy, illustrated in Figure 4. With
3The selected dialogue samples are rated above 90 for naturalness and consistency on a 100-point scale.
closer inspection, the model tuned on SalesBot 1.0 favors selecting the transition policy in nearly half of the cases, whereas the model tuned on SalesBot 2.0 tends to continue with chit-chat rather than transitioning. This behavior corresponds with the characteristics of the datasets outlined in Table 4. Although fewer transitions may result in missing potential business opportunities from a sales perspective, maintaining the conversation could be crucial for fostering a non-aggressive approach in various sales contexts.

heading: 5.2 Dialogue-Level Evaluation
text: To facilitate dialogue-level outputs, we employ llama-2-7b-chat, initiating multiple user simulators with 50 distinct personas generated by ChatGPT as system prompts. Each persona includes randomly specified user preferences for predefined intents such as no_preference, not_interested_2, not_interested_4, not_interested_all, guiding the simulator to reject topic transitions when the topics do not align with the user’s interests. An example of a system prompt for the user simulator can be found in Appendix D: Subsequently, our SALESAGENT engages with each user simulator five times, yielding a total of 250 dialogues.
Table 5 displays average number of turns and proceed TOD rate for each model. Notably, SalesBot achieves a 100 percent TOD rate, implying a potential tendency of the model to “aggressively” direct conversations towards task-oriented dialogues customers may not like in many scenarios.
We use GPT-4 to evaluate these 250 dialogues, assessing them across five criteria: naturalness, coherence, smoothness, agent consistency, agent aggressiveness. The evaluations, detailed in Table 5,
show that the model trained on our proposed dataset yields significantly more natural and coherent dialogues compared to its predecessor. Moreover, the model trained on SalesBot 2.0 demonstrates a nearly 50 percent improvement in consistency and a 45 percent reduction in aggressiveness compared to SalesBot 1.0. Additionally, SalesBot 2.0 excels in smoothly transitioning dialogue topics, marking a substantial enhancement in real-world application viability.

heading: 6 Conclusion
text: This paper introduces a novel framework for generating intent-oriented dialogues, leveraging the commonsense knowledge of large language models (LLMs). Our proposed SalesBot 2.0 dataset encompasses thousands of human-like dialogues, demonstrating smoother transitions, improved naturalness, and enhanced consistency compared to existing datasets. This dataset represents a substantial contribution to the advancement of more sophisticated and effective conversational agents capable of aligning with users’ preferences and requirements. Additionally, we develop a model, SALESAGENT, through fine-tuning on the proposed dataset. Employing the innovative chain-of-thought (CoT) reasoning technique, the model not only generates human-like responses but also formulates thoughts about the user’s potential intent and determines the response policy. In the evaluation phase, our model consistently outperforms its counterpart trained on previous datasets, SalesBot 1.0, across various perspectives. We also find out that the model without tuning lacks the ability to understand complex scenarios and react properly with mere instructions in the system prompt. This work serves as inspiration for future research in this domain, paving the way
for the development of more intelligent conversational agents capable of better understanding and responding to users’ needs.

heading: Limitations
text: While SalesBot 2.0 showcases higher quality and natural language capabilities, it has certain limitations. The pipelines and evaluation relies solely on LLMs, making the generated data and evaluation score highly dependent on the LLM’s quality. This reliance introduces the risk of noisy or inaccurate output if LLMs misunderstand instructions. Additionally, prompt design is critical as LLMs are prompt-sensitive, impacting output quality and accuracy. Careful prompt design is essential for desired outcomes. Future research could explore alternative prompt designs, enhance the data generation framework, and construct more robust and higher-quality datasets to address these limitations.

heading: Acknowledgements
text: We thank the reviewers for their insightful comments. This work was financially supported by the National Science and Technology Council (NSTC) in Taiwan, under Grants 111-2222-E-002013-MY3, 111-2628-E-002-016, and 112-2223- E002-012-MY5. We thank to National Center for High-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources.

heading: A Hyperparameters
text: We fine-tune our SALESAGENT on llama-2-7b-chat with the script of FastChat4. Here is the detailed hyperparameters:
4https://github.com/lm-sys/FastChat
• epoch: 10 • batch size: 1 • gradient accumulation steps: 32 • learning rate: 0.00002 • warn up ratio: 0.04 • learning rate scheduler: cosine • model max length: 1024

heading: B Prompt Templates for SalesBot 2.0
text: In the development process of the SalesBot 2.0 dataset, we employ an existing LLM to refine and enhance the previous dataset, SalesBot 1.0. Here we present the prompt templates used within this framework.

heading: B.1 Chit-Chat Dialogue Revision
text: Given that SalesBot 1.0, developed using BlenderBot, occasionally results in unnatural interactions, this section aims to enhance the fluency and naturalness of open-domain chit-chat dialogues. To improve the quality, we instruct an LLM to identify and focus on inconsistent interactions, subsequently modifying them to achieve greater coherence and smoothness.
You will be given a conversation between two people. Here is what you should do: 1. Identify the inconsistent utterances. 2. Give some reasons why they are
inconsistent. 3. Modify the dialogue based on previously
identified utterances. 4. The rewritten dialogue should be more
than 6 turns. Here is the conversation: <Dialogue> You MUST follow the format as : <output_format>

heading: B.2 Potential Intent Detection
text: This section focuses on identifying the implicit intent of the user to determine the suitable timing for transitioning to a task-oriented scenario.
You will be given a dialogue and a list of topics of conversation. Please tell me which of the following topics will be the most reasonable one to be pivoted to in the dialogue.
Here is the dialogue: <Dialogue> Here is the list of topics: <Intent List> NOTE: 1. You MUST choose one of the above topic. 2. DONOT create any topics that are not
listed above. 3. You should choose the one that is the most
related to the topic. The output format should follow the below format: <output_format>

heading: B.3 Dialogue Continuation
text: Previous research (Chiu et al., 2022b) highlighted that SalesBot 1.0 was perceived as overly aggressive by human evaluators. This section aims to refine the transition process to make it more natural and comfortable for customers, thereby reducing the perceived aggressiveness of the agent and ensuring smoother dialogue transitions.
Here is the potential intent (with description) and an incomplete dialogue:
<Intent> <Dialogue>
Your goal is as following: 1. Continue the dialogue with reasonable
responses considering the previous context.
2. Continue the dialogue with topics that implicitly related to the intent listed above. 3. If you found it hard to transit, please find other topics related to the contexts and intent and chat for several turns before the final transition. 4. Continue the topic if it's not yet close to the end. 5. For each topic, please generate at least 5 turns. 6. The agent should pivot the conversation smoothly, which means the transition involved longer conversation. 7. The user should then somehow mention the given intent, after the pivoted dialogue. 8. Use more reasonable phrases to transit the topic of the conversation. 9. End the dialogue with task-oriented style (TOD) where the agent fulfills the user's intent listed above.
Please note that both the user and agent should not explicitly disclose the intent; instead, the dialogue is naturally guided to the potential purpose. ----- Output should follow the format below: <output_format>

heading: B.4 Transition Boundary Detection
text: This section concentrates on identifying the precise moment when the agent should handover the conversation to the task-oriented dialogue (TOD) bot. The objective is to effectively guide the transition from casual chit-chat to a more focused, taskoriented dialogue.
You will be given a dialogue below and a potential intent below: <Intent> <Dialogue> and your goal is as following: 1. Identify the first utterance that
apparently mentions the intent given above.
2. You should choose only one turn in the given dialogue. 3. The chosen turn should be said by User. Please follow the output format as below: <output_format>

heading: C Prompt for the Untuned Baseline
text: In our experiments, we compare our refined, tuned model against an untuned LLM. This prompt directs the llama-2-7B-chat model to operate as a salesperson without any further modifications or tuning.
Here is a list of potential intents that might be referred by the user: {intents} Think carefully to determine the potential intent and provide suitable response given the above dialog history. You should response as a real conversation. If you think user has explicitly mentioned the above intent, you should say \"Proceed to task oriented dialog agent.\"

heading: D Persona Prompt Template
text: In our experiments, we conduct dialogue-level evaluations, necessitating an environment where agents can interact effectively. Below, we present the prompt template used to instruct the model to simulate a user with a diverse range of personas.
You are not an AI. {persona} You are not interested in {intents} if the agent ask any about one of them, do not ask for any recommendations and you should say, \"I don't want to talk about this. Let's talk about something else\". Imagine you are a real person. You are having chat with a online agent, so the repsonse do not include any expresssions. Remember, maintain a natural tone. Your response should be only your text response without any other expressions. Keep it as short as possible.

heading: E Evaluation Prompts

heading: E.1 Schema
text: Definition of the scores: - Naturalness (the higher the more natural): The content of the dialogue is in general natural and human-like. - Coherence (ther higher the more coherent): The dialogue is coherent and easy to follow. - Smoothness (the higher the smoother): The dialogue is smooth and the agent is able to smoothly steer the conversation toward a certain topic without explicit transition. - Agent aggressiveness (the higher the more aggressive): The agent is considered to be aggressive if it is too pushy and it change the topic without the user's intention. Especially, if it directly proceed to task-oriented dailogue agent or directly provide user intents options without the user's explicit intent, the agent is very aggressive. - Agent consistancy (the higher the more consistant): The agent is considered as consistant if it is able to maintain the same personality and topic throughout the conversation. { "naturalness": {
"reason": "<reason for naturalness score>", "score": <naturalness score> },
"coherence": { "reason": "<reason for coherence score>", "score": <coherence score> }, "smoothness": { "reason": "<reason for smoothness score>", "score": <smoothness score> }, "agent aggressiveness": { "reason": "<reason for agent aggressiveness score>", "score": <agent aggressiveness score> }, "agent consistancy": { "reason": "<reason for agent consistancy score>", "score": <agent consistancy score> } }
E.2 Instructions
The following is a conversation between a user and a salesbot, and the goal of salesbot is to smoothly direct the conversation toward a certain topic and proceed to task-oriented dialogue agent. Score the following dialogue generated on a continuous scale from 0 to 100. Dialogue: {dialog} Format: {eval_schema} Output:

heading: F Dialogue Examples
text: We present sample dialogues generated by our SALESAGENT models, tuned on SalesBot 1.0 and 2.0, as well as an untuned model, displayed in Tables 6, 7, and 8 respectively. The dialogue generated by the model tuned on SalesBot 2.0 is notably more coherent and smooth than that of the model tuned on SalesBot 1.0. This improvement demonstrates that SalesBot 2.0 significantly enhances the model’s ability to guide users towards target tasks and serves as an effective resource for training models to perform salesperson roles.

heading: SALESAGENT tuned on SalesBot 1.0

heading: SALESAGENT tuned on SalesBot 2.0

heading: Llama-2 w/ only instruction

heading: Future Work
text: Future research could explore alternative prompt designs, enhance the data generation framework, and construct more robust and higher-quality datasets to address these limitations