TOAD: Task-Oriented Automatic Dialogs with Diverse Response Styles (2024)

Abstract: In light of recent advances in large language models (LLMs), the expectations for the next generation of virtual assistants include enhanced naturalness and adaptability across diverse usage scenarios. However, the creation of high-quality annotated data for TaskOriented Dialog (TOD) is recognized to be slow and costly. To address these challenges, we introduce Task-Oriented Automatic Dialogs (TOAD), a novel and scalable TOD dataset along with its automatic generation pipeline. The TOAD dataset simulates realistic app context interaction and provide a variety of system response style options. Two aspects of system response styles are considered, verbosity level and users’ expression mirroring. We benchmark TOAD on two response generation tasks, and the results show that modeling more verbose responses or responses without user expression mirroring is more challenging.1

heading: 1 Introduction
text: Task-Oriented Dialog (TOD) stands as a fundamental task of machine intelligence (Young, 2010), involving goal-driven conversations between a human and a system to achieve specific tasks. It has been used in training virtual assistant systems for various real-world applications, including mobile phones (Hoy, 2018), virtual reality devices (VR) (Kottur et al., 2021), smart home devices (Duong et al., 2019), online shopping assistants (Yan et al., 2017), and trip booking helpers (El Asri et al., 2017b).
Advancements in developing TOD systems face a challenge posed by conflicting requirements (Hu et al., 2023): the demand for extensive datasets clashes with the substantial time and financial investments (months if not years) required for data collection (Larson and Leach, 2022). Looking
∗Work done during an internship at Apple. 1The data and code are available at https://github.com/
apple/ml-toad.
ahead to the next generation of virtual assistants, an expectation arises for them to adapt their response styles to different usage scenarios, e.g. if equipped with a screen, ensuring a sustained level of naturalness and communication efficiency and ultimately enhancing the user experience. However, existing datasets lack consideration for adaptive response styles and neglect to simulate interactions with app contexts like calendars or alarms.
This paper presents Task-Oriented Automatic Dialog, TOAD, a highly automatic generated dataset, along with its generation pipeline, designed to address the challenges mentioned above. In TOAD, we not only simulate realistic interactions with app context and diverse conversational phenomena, but also analyze two aspects of response style—verbosity level and users’ expressions mirroring, aiming to enhance naturalness and adaptiveness for various usage scenarios. As shown in the dialog example in Fig. 1, for each turn of system utterance, we provide a spectrum of six responses style options. The dataset is constructed using an
8341
automatic generation pipeline, leveraging the zeroshot generation capability of the latest Large Language Models (LLMs), such as ChatGPT2. Due to its automatic nature, the TOAD dataset is scalable in terms of data size and service coverage. To provide a benchmark for future studies, we establish reference scores for two TOD Natural Language Generation (NLG) tasks, evaluating a variety of baseline models. The results indicate that modeling more verbose or responses without user expression mirroring is more challenging.
The contributions of this work are three-fold: 1) A new TOD dataset with multiple response styles and realistic app context interaction, such as revising or deleting calendar events. 2) TOAD is the first that investigates the naturalness and adaptiveness of system response, providing insights into the dimensions of style that virtual assistants should consider. 3) An automatic TOD data generation pipeline for scalable, cost-effective expansion of data size and domains.

heading: 2 Related Works

heading: 2.1 Task-Oriented Dialog Dataset
text: To our best knowledge, TOAD stands out as the first highly automatically generated TOD dataset. A concurrent work, LUCID (Stacey et al., 2024), also adopts a similar LLM-driven approach for dialog generation. All previous datasets relied on human annotations or paraphrasing, which makes the collection process costly and time-consuming. Moreover, our dataset also admits all of the unique properties listed in Section 1.
Previous TOD datasets mainly fall into two categories based on how dialog utterances are collected: Machine-to-Machine (M2M) and Wizardof-Oz (WOz). In the M2M setup, proposed by Shah et al. (2018), systems or schemas exhaustively simulate dialog with skeleton plots, represented in structured formats. The dialog plots are realized
2https://openai.com/blog/chatgpt
into natural language by crowd workers. A representative dataset is the SGD (Rastogi et al., 2020), which consists of dialogs across 16 domains. They defined a list of meta information as schema, such as valid slots and supporting intents for each domain and utilize the slot-filling strategy to simulate the dialog plots. Another dataset, STAR (Mosig et al., 2020), defined explicit ideal dialog flows for each domain and simulated realistic user behaviors such as small-talk interruptions.
Another line of TOD data collection setup, the WOz (Kelley, 1984), employs crowd workers playing roles both of the user and the system, to directly produce utterances in a more improvised manner. The user is provided with an overall goal to achieve throughout the conversation, while the “system” needs to respond with access to a database based on the user’s preference. Such WOZ set-up has been successfully validated by WOz (Wen et al., 2017) and FRAMES (El Asri et al., 2017a). A popular dataset MultiWOZ (Budzianowski et al., 2018) designed a user-friendly interface for the Wizards and easy-to-follow user goals, resulting in diverse and semantically rich data. The recent Multi3WOZ (Hu et al., 2023) expands this paradigm, collecting a large-scale multilingual TOD dataset with parallel utterances in four languages over the same conversational flows.

heading: 2.2 Dialog Distillation
text: Recent advancements in LLMs have demonstrated their improved ability to follow human instructions and generate fluent conversation utterances for a range of understanding and generation tasks. A list of recent works embrace this concept, extracting knowledge and dialog from LLMs to train student or in-domain models. SODA (Kim et al., 2023) distilled social interactions dialogue by contextualizing commonsense knowledge from a social knowledge graph. PLACES (Chen et al., 2023a) generated a multi-party social dialog by prompting the OPT (Zhang et al., 2022) model with topic, facts,
and dialog as in-context learning examples. MathDial (Macina et al., 2023) distilled teacher-student math reasoning tutoring dialog through pairing human teachers with a LLM. There are also works that distil multi-turn open-domain instructional conversations, such as Baize (Xu et al., 2023) and UltraChat (Ding et al., 2023). Our TOAD represents the first work in distilling TOD dialog with multiple response style options, which requires more nuanced control over the conversation content.

heading: 3 Data Generation Pipeline
text: The TOAD pipeline, depicted in Fig. 2, comprises three key stages for the automated creation of dialogues. The initial stage involves the generation of device context, simulating users’ personas and their device app contexts. In the second stage, essential conversation attributes, including conversational content, flow, and language phenomena, are determined and incorporated into a dialog plot. The final stage involves the realization of the dialog plot into dialog utterances by a dialog generator. This generator simulates user speaking habits and provides a spectrum of system response styles.

heading: 3.1 Persona-Grounded Context Generation
text: Interacting with on-device context information such as calendar and alarms is an important part of real-life virtual assistant use-cases. A recent dataset, PRESTO (Goel et al., 2023), provides simulated structured contexts, but their influence is very limited as most dialogs are irrelevant to their contexts and there is no interaction such as modification or deletion.
Data diversity is important to the model’s robustness and generalization. However, LLM-generated data given the same input prompt often lacks diver-
sity. To address the two issues mentioned above, we develop a persona-grounded context generation pipeline, which combines sampling from external data sources and chain-of-thought generation (Wei et al., 2022). Recent work (Hu and Collier, 2024) has shown that LLMs are capable to simulate different perspectives.
We synthesize each persona in 3 steps: 1) We synthesize random occupation information, by sampling from work statuses including employed, unemployed/retired, and student. For students or the employed, occupations are based on NAICS 6-digit industries.3 Additional occupation details such as location, affiliation, and job level are generated by prompting the LLM. 2) We sample surname and race together from the 2010 US Census data,4 and sample other attributes such as gender, MBTI personality from pre-defined sets. 3) The sampled information is input into the LLM to write an introduction with additional fictionalized details (e.g. first name, age, marital status, hobbies).
Based on each persona’s introduction, attributes and a random ‘current time’, we prompt the LLM to generate app context instances for each service. Details about the supporting services are provided in Section 5. There are dependencies across services, e.g. information of generated contacts might be shared by messages and calendar.

heading: 3.2 Plot Generation
text: Schema Our pipeline, inspired by the approach in SGD (Rastogi et al., 2020), establishes a schema for each service by outlining supported intents, associated slots and relevant meta-data. For example,
3https://www.census.gov/naics/2022NAICS/2022_ NAICS_Structure.xlsx
4https://www2.census.gov/topics/genealogy/ 2010surnames/names.zip
the calendar_events service has four supporting intents (create, modify, check and delete). Each intent specifies required slots like start_time, date and duration along with optional slots such as attendees, name and location. Additionally, we also specify a list of meta-indicators for each intent to control the conversation flow. For instance, the intent of reserve_table is marked as for “require_confirmation=True”, indicating the need for an additional confirmation in the dialog flow. Further details about the schema are provided in Appendix A.
Dialog Phenomena To enhance the authenticity of our conversation flows, we simulate various dialog phenomena, exemplified in Tab. 2. These phenomena are categorized into multi-intent and single-intent types. Compound and compositional dialogs fall into the multi-intent category, where users pursue related or unrelated goals simultaneously or their initial intent depends on another implicit intent. In the Tab. 2 example, to obtain the weather information for the day of the “Art Class”, an implicit intent check is needed to get the actual date of the “Art Class”. Self-correction and complex referral are phenomena that simulate real-life conversation dynamics. In self-correction, users correct themselves or alter their requests midsentence or afterward. In complex referral, users employ realistic referring expressions that require additional logic to comprehend. For instance, an oral expression like “earliest alarm” requires ranking all alarms in app context by time to pinpoint the user’s reference accurately.
Intent Sampling In TOAD, intent represents user’s intention and determines the conversation content and goal. To compose the dialog, we sample services and intents, and generate corresponding slot values through LLM prompts. For consis-
tency, all slot values are generated simultaneously.
For multi-intent compound dialog, two intents and their respective slot values are independently sampled. While for compositional dialog, as the output of the implicit intent will be used as the input of the initial intent, the two intents are jointly sampled with a slot matching strategy. Further details on the intents sampling and slot values generation are provided in Appendix C.
Plot Representation As illustrated in Fig. 2, a plot serves as a conversation skeleton, consisting of a sequence of Meaning Representations (MRs) that encapsulate all the necessary dialog information. The manipulation of these MRs allows control over the dialog’s content and flow. In contrast to SGD and STAR, which rely on a finite set of functionlike actions for MRs, TOAD opts for a pseudocode format to represent actions. This choice avoids dependence on a fixed set of predefined actions, enriching the meaning space, particularly in terms of referring expressions. Furthermore, the decision to use pseudocode aligns with the fact that most latest LLMs have been pre-trained on code, ensuring their comprehension of code-like MRs.
Plot Construction Dialog plot construction follows the flowchart in Fig. 3. The core mechanism employs a slot-filling strategy, with the system asking the user to fill all required slots to achieve their intent. The meta-indicators specified in the intent schema also determine the plot components, e.g. including additional confirmation turns or replying with search result summarization. For multi-intent dialogues, the plots are composed by merging individual plots for each intent in a natural order. The details of the merging policy are provided in Appendix D.
Context Interaction Dialogues involving interactions with context apps are ubiquitous in real-life scenarios but have not been adequately addressed in existing datasets. TOAD is the first work that includes conversations with context interactions, supporting generic intents of checking, deleting, and modifying on app services such as calendar events, alarms, and messages.
This expansion significantly broadens the scope for simulating dialogues with natural referring expressions and introduces the concept of proper referring expressions. Users can refer to a calendar event using any slots, but the system should adhere to a preference order: While multiple slots may be unambiguous, certain slots are more intuitive for users. For instance, when referring to a calendar event, “the 9 a.m. meeting” is preferred over “the meeting in the office.” Therefore, referring by time is prioritized over referring by location. For each intent, we define a preference order for the slots, enhancing the naturalness of the system response.
Database query Traditional TOD datasets support intents that require database queries, such as restaurant or movie searching. In contrast to SGD’s use of FreeBase for realistic slot values, our TOAD pipeline leverages LLMs as a knowledge database. To ensure consistent query results, we prompt the LLMs with input slot values as conditions. For each query, we instruct LLMs to generate five different instances, simulating the search operation and ensuring diversity in the returned values. The advantages of using LLMs as a knowledge base
include: 1) It avoids the issues of empty or noisy searching results. 2) The query outcomes are more consistent with other slot values.

heading: 3.3 Dialog Generator
text: The dialog generator realizes the composed plots into natural utterances. Similar to SGD and STAR, we adopt the Machine-to-Machine (M2M) approach, where different LLMs take turns to play the roles of user and system for generating conversation utterances.
Style Control To offer natural responses across various scenarios, we generate a spectrum of response styles for each system turn as shown in the example in Fig. 1, including combinations of low, mid and high verbosity and mirroring or no mirroring. This is achieved by providing In-Context Learning (ICL) examples and style definitions for different styles to LLMs. The motivations behind these style designs are explained in Section 4.
Given the six style options for each system turn, a selection must be made to construct the dialog history, which will shape the subsequent generation. The dialog history significantly influences the generation of future conversation utterances. For instance, if the system consistently responds with low verbosity, user utterances tend to become shorter as well. Hence, the most natural style for a given conversation context should be selected, referred to as the default style, which should balance between efficient communication and minimizing the risk of misoperation. The selection rules are explained in detail in Section 4.
Prompt Template The prompt templates for user and system encapsulate three key pieces of information: 1) Current turn dialog action, 2) dialog history, and 3) style instruction. ICL demostration for action-to-utterance are also provided to guide the utterance realization. For user turns, the style instruction describes the user’s speaking habit based on the sampled persona. For system turns, the style instruction provides a brief definition of each style with corresponding ICL examples. Examples of prompt templates can be found in Appendix H.
For efficiency and cost considerations, TOAD prompts LLMs to generate all system styles in a single inference pass. While generating one style utterance at a time improves textual quality and control performance, it represents a trade-off between quality and cost.

heading: 3.4 Dataset Quality Control
text: To ensure the quality of the generated dataset, we conducted a two-step quality control process. The first step involved two automatic sanity checks: 1) Removing datapoints with utterances containing unformatted values, such as internal datetime representations like ‘2025-12-16’. 2) Removing datapoints that have missing slot values in the corresponding utterances.
The second step addressed the potential inconsistencies between plot actions and utterances that may arise due to the non-deterministic nature of language models. Although human evaluation is considered the gold standard, it is not always feasible or scalable for large-scale dataset expansion. As an alternative, we propose using a language model evaluator to perform quality control, an approach that has gained popularity in recent works (Chen et al., 2023b; Zeng et al., 2023; Liu et al., 2024). These studies have demonstrated the effectiveness of language model evaluators in assessing various aspects of generation tasks, such as summarization and data-to-text.
To implement this approach, we provide the LLM evaluator (gpt-3.5-turbo) with evaluation criteria, app context, action plot, and corresponding utterances. The evaluator then determines whether the utterances are consistent with the action plot and the app context. In our case, the LLM labeled 37 data points as inconsistent. Upon manual examination, we confirmed that 32 of these samples were indeed inconsistent. We listed the template used for quality control in Appendix I.

heading: 4 System Response Style
text: The primary evaluation criterion for today’s virtual assistants is their accuracy in responding and assisting users in achieving their goals. The next generation of virtual assistants is expected to exhibit increased naturalness and adaptability to various usage scenarios, which means the systems should be able to respond with different styles. TOAD stands as the first work in investigating the dimensions of style that virtual assistants should consider. We aim to open the discussion on optimal response styles to encourage future exploration. In this work, our focus centers on two main aspects: verbosity level and mirroring.

heading: 4.1 Verbosity
text: Verbosity refers to the degree of details or amount of information expressed in a response utterance. In many cases, clear and concise communication is preferred as it ensures efficient delivery and easy comprehension. However, certain contexts may demand a higher verbosity to provide comprehensive explanations and avoid ambiguity. It’s essential to strike a balance and use an appropriate level of verbosity based on the specific communication situation and user needs.
In TOAD, we define three levels of verbosity, illustrated through examples in Fig. 1: Low verbosity responses should only contain a few words and may not necessarily be a complete sentence. Mid verbosity responses should use pronouns to refer any previously mentioned events. High verbosity responses should refer to events with proper and detailed expression and provide all relevant information. System verbosity critically impacts user experience, and we propose a set of rules to determine the most natural response for a given context, which we consider the default style.
1. If there is special language phenomena in the user’s last query, the system should employ higher verbosity. 2. If the system needs to respond with new information, verbosity should be mid or high. 3. If the system is requesting user confirmation for an irreversible operation, such as calender deleting or ticket booking, verbosity should be high. 4. Otherwise, the system should exhibit low verbosity for efficient communication.
We note that this approach is subjective, and different styles may be favored in various situations.

heading: 4.2 Mirroring
text: Extensive research (Liao and He, 2020; Vartanov et al., 2023) demonstrates the benefits of mirroring user behavior, emotions, personas, and expressions in psychological tasks. Mirroring enhances mutual understanding, promotes efficient communication, and consequently leads to improved user feedback (Garrod and Pickering, 2009). In TOAD, the response style of mirroring refers to the reuse of noun or verb phrases from the user’s expression whenever feasible, as illustrated in Fig. 1. In contrast, a non-mirroring style avoids directly copying the user’s expression.
Mirroring is generally a favorable strategy, fos-
tering natural conversation. However, caution is warranted, as a virtual assistant should consistently maintain professionalism and ensure positive communication. Instances where mirroring may be inappropriate include:
• Emotional and improper phrasing: In real-life scenarios, users may use sentimental or inappropriate language, such as cursing words. • Bias: Users’ expressions may contain negative connotations related to gender, culture, race, or politics. A virtual assistant should avoid reinforcing these biases. • Non-factual information: Users may not always provide accurate information, particularly in knowledge domains. For instance, asking “Who is the king of the United States?”. • Ambiguous terms or referring expressions: Mirroring potentially ambiguous expressions could be confusing to the user. For example, a user might mention “my meeting with David,” and if the system copies this referring expression, the user could be uncertain when identifying the event, as they may not be aware of the possibility of having more than one scheduled meeting with David.
Mirroring user expressions is not universally safe or appropriate, and it is challenging to exhaustively enumerate all misuse cases. The outlined situations provide a foundation for identifying potential challenges, and we encourage future exploration.

heading: 5 Dataset Statistics
text: The current TOAD dataset covers 11 service domains with 37,678 turns (18,839 user turn and 113,034 system response variations, 131,873 turns in total) in 8,087 dialogues. It’s worth highlighting that as TOAD operates as an automatic generation pipeline, both datasize and covered service can be scaled up. Additionally, the creation of a new service schema typically requires only 15 minutes.
Across the 8087 dialogs, 40.5% are multi-intent compound, 20.9% are compositional and 38.6% are single intent. The distributions of the number of turns and word counts in utterances are shown in Fig. 4(i) and (ii) respectively. As for local phenomena, 19.0% dialog are self-revision, 15.1% have complex referral, 3.3% have both and 62.6% have no local phenomena. Note that local phenomena can appear in both single and multi-intent dialog.
Fig. 4(iii) illustrates the service distribution which is relatively uniform for single intent and compound dialogs. However, in compositional dialog, due to the joint sample requirement, certain services may have less matchable slots. Scaling up the supporting services in the schema can potentially expand the joint service sample pool.
As mentioned above, TOAD does not have a fixed set of plot actions. Fig. 4(iv) displays the distribution of actions whose occurrences are greater than 3%. Based on the selection rules for the default response style, 61.7% of the system responses are default high verbosity, 28.8% and 9.5% are mid and low verbosity. The average word counts for
these three levels are 17.6, 9.0 and 2.5 respectively.

heading: 6 Evaluation
text: TOAD data supports a range of TOD tasks, including Intent Detection, Slot Labeling in Natural Language Understanding, Dialog or Action State Tracking (D/AST) and Natural Language Generation (NLG). This paper establishes benchmarks for two response generation setups, aligning with TOAD’s primary objective of producing more natural, realistic dialogues and diverse response styles.

heading: 6.1 Response Generation Benchmarks
text: The first benchmark setup is a traditionally surface realization setup, where a dialog action is aimed to convert into a natural language utterance (Williams and Young, 2007) The second, following Hu et al. (2023), is ‘oracle’ language model setup, where both previous dialog history and current action are available to produce the target utterance. For both setups, the target verbosity and mirroring option are given as part of the input.
We partition the TOAD dataset into three sets: a training set (≈ 81.8%), a test set (10%), and a zero-shot test set (≈ 8.2%) containing dialogs exclusively related to the ‘banking’ service. The zeroshot test set aims to assess models’ generalization ability to unseen services.
Each response style option is treated as an individual datapoint. The surface realization setup only includes non-mirroring response options, as it does not have access to previous dialog utterances. Whereas, the oracle language model setup considers all six options. In total, the two setups have 56.5K and 113K datapoints respectively
We applied supervised fine-tuning (SFT) to five baseline models from two LM families: Encoderdecoder models, including FlanT5-(base, XL and XXL) (Chung et al., 2022), with parameter sizes of
250M, 3B and 11B respectively, and decoder-only models Llama2-(7b, 13b) (Touvron et al., 2023). The SFT hyper-parameters are provided in Appendix F.
We evaluate model predictions using three widely-used NLG metrics: BLEU (Papineni et al., 2002), ROUGE-L (Lin and Hovy, 2002) and METEOR (Banerjee and Lavie, 2005) scores. The results are reported in Tab. 3. In both setups, we observed that performance is generally better on the test set compared to the zero-shot test set, and increasing model sizes can improve the performance on both sets. However, the performance gaps between the test and zero-shot test sets become smaller for LMs with increasing sizes, which we believe is due to the increasing zero-shot generalization ability of larger LMs.
Additionally, we consistently observe that the oracle language model setup outperforms the surface realization setup, which means having extra information of the dialog history can steadily improve the response generation performance.

heading: 6.2 Predictions on Different Response Styles
text: In this section, we explore the modeling difficulty posed by different response styles. In Tab. 4, we compare the performances on mirroring and nonmirroring responses as well as the performances
across different verbosity styles. All results are evaluated on the predictions generated by the oracle language model setup on the test set.
We consistently noting that the performance on mirroring responses is better than non-mirroring ones, as expected, given that mirroring allows the system to replicate user expressions directly from the dialog history, making it easier to predict. We also observe that as verbosity increases, performance tends to decrease. This decline is likely attributed to lower verbosity resulting in shorter utterances with a reduced vocabulary, making them easier to model.
Another observation worth noticing is that, even for non-mirroring options, having access to dialog history can yield improved modeling performance, as evident when comparing the left-most column of Tab.3 with the second left-most column of Tab.4.

heading: 7 Conclusion
text: In conclusion, our study explores the naturalness and adaptiveness of system responses for the next generation of TOD virtual assistants. We introduce TOAD, a dataset designed to train TOD systems for diverse verbosity levels, mirroring styles, and realistic app context interactions. Additionally, we present a cost-effective and scalable automatic data generation pipeline as a practical alternative to traditional human annotations. By addressing those critical gaps, we aim for TOAD to inspire future exploration in modeling and analyzing system response styles.

heading: 8 Limitations
text: Entities The names and entities are generated by ChatGPT, therefore some of them are figures from publications, such as movies or novels.
Difficult cases for style modeling When producing the six response styles, certain scenarios pose challenges to model desired styles, which means the difference between some styles might become less evident. We listed some observations during TOAD’s construction:
• In cases when the system needs to present new information to the user, such as reporting database search results, in order to provide necessary information, sometimes low verbosity option can be relatively lengthy. • When the system is requesting value for a new slot, it usually struggles to mirror users, who
may not have mentioned the required slots. • The use of proper referring expressions in user
interactions with contextual apps can result in responses that are very similar between mirroring and non-mirroring styles.

heading: 9 Ethics and Risks
text: To ensure no user is disadvantaged, a dataset for model training needs to represent a diverse range of users. The distribution of persona attributes in the TOAD dataset is detailed in Appendix E.

heading: A Schema Details and Example
text: The schema is defined in JSON format. For each service, there are a list of supporting intents and a list of slots. For each intent, there are lists of “required_slots”, “optional_slots” and “result_slots”. There are also meta-indicators to control the plot construction for that intent. {
"service_name": "calendar_events", "intent_operations": [
{ "name": "create", "description": "Create a calendar
event.", "require_input_values": true , "require_context": false , "require_confirmation": false , "return_list": false ,
"report_result": false , "check_on_input": false , "can_refer_to_input_slot": true , "minimum_input_slot_number": 1, "minimum_initial_slots": [], "summary_emphasis_slots": [], "required_slots": [
"date", "start_time", "duration_time"
], "optional_slots": [
"attendees", "name", "location"
], "result_slots": []
} ], "slots": [
{ "name": "date", "description": "The date of the
calendar event.", "potential_values": [], "alias": ["start_date"]
}, {
"name": "start_time", "description": "The time of the calendar event.", "potential_values": [], "alias": ["time"]
}, {
"name": "duration_time", "description": "The duration_time of the calendar event.", "potential_values": [], "alias": []
} ]
}
Listing 1: Schema example for intent "create" of service "calendar_events" with parts of the relevent slots.

heading: B Data Structure
text: Each datapoint contains a multi-turn dialog action plot and corresponding utterances, each system turn contains 6 styles of responses and an indicator of the default style. Each datapoint also contains a user persona, relevant app contexts and labels for services, intents and phenomena.
C Intents and slot values sampling
The intent sampling for single-intent dialog is straightforward. A service is randomly sampled first and then one of the supported intents is randomly sampled. For compound dialog, two intents are sampled by this method.
As for compositional dialog, intents are sampled based on the slot matching strategy, because the
output slot of the inner intent, sinneroutput, will be used as the input slot of the outer intent, souterinput. As shown in the compositional example in Tab. 2, the output slot of the inner intent “check” is “date”, which is passed to the outer intent “get_weather” as the input slot. The slot matching strategy does not require sinneroutput and s outer input to be exactly the same, instead, it only requires souterinput being a super-set of sinneroutput. For example, in the phrase “Book an alarm at the time when the show begins,” the souterinput =“time” is the super-set of the sinneroutput =“showtime”. As shown in the schema example in Listing 1, for each slot, its parent slot is assigned in “alias”.
To ensure the consistency among the slot values, all values are generated by LLM within one inference. The input slot values are generated first, such that the output slot values can condition on them.

heading: D Multi-intent plot merging
text: Compound dialog contains two intents. When constructing the plot for compound dialog, the plots for two individual intents are merged into one, but with certain merging rules to maintain the naturalness of the conversation flow. Here are the merging policy:
• Initial user query: Concatenate the actions of two individual initial queries and realize them into one single user utterance. • For system response: If both system responses require user information, arrange the responses into two subsequent turns. Otherwise, combine two system responses into a single turn, but re-ordered to keep confirmation responses first. • For subsequent user turn: Only answer the system response.
As for compositional dialog, the initial user queries will be merged by substituting the matching input slot of the outer intent with the inner intent action, as illustrated in Tab.2. Then the rest of the plots will be rearranged that the system should always finish the plot for the inner intent first.

heading: E Persona and App Context Statistics
text: In TOAD, we simulate 500 personas with diverse backgrounds, with one persona randomly assigned to each dialog as the user’s information source. The gender distribution of the simulated personas is approximately 52% male and 48% female.
Examining the racial distribution, we find that 17% are White, 10% Hispanic or Latino, 6% Black or African American, 4% Asian or Pacific Islander, 4% American Indian or Alaska Native, and 59% are not specified. The job descriptions encompass a total of 67 distinct occupations categorized into four job levels: senior (18%), intermediate (31%), entry-level (22%), and not-specified (29%).
In terms of context app distribution, the dialogs are diversified with 17.9% involving calendar events, 13.5% messages, 13.5% reminders, 13.1% alarms, 8.9% contacts, and 33.1% having no specified context.

heading: F SFT hyperparameters
text: SFT batch sizes for FlanT5-250m, 3b, 11b are 16, 16, and 4, respectively. Llama2-7b and 13b have batch sizes set at 2. The learning rates are 3e − 5, and fine-tuning halts upon meeting the early stopping criterion, which is 5 consecutive increases in validation set loss.

heading: G Plot and Dialog Examples
text: We show more dialog and action plot examples in Tab. 5.

heading: H Dialog Generator Template

heading: I Quality Control Template

heading: Future Work
text: By addressing those critical gaps, we aim for TOAD to inspire future exploration in modeling and analyzing system response styles