NATCS: Eliciting Natural Customer Support Dialogues
Year: 2023

Abstract:
Despite growing interest in applications based on natural customer support conversations, there exist remarkably few publicly available datasets that reflect the expected characteristics of conversations in these settings. Existing task-oriented dialogue datasets, which were collected to benchmark dialogue systems mainly in written human-to-bot settings, are not representative of real customer support conversations and do not provide realistic benchmarks for systems that are applied to natural data. To address this gap, we introduce NATCS, a multi-domain collection of spoken customer service conversations. We describe our process for collecting synthetic conversations between customers and agents based on natural language phenomena observed in real conversations. Compared to previous dialogue datasets, the conversations collected with our approach are more representative of real human-to-human conversations along multiple metrics. Finally, we demonstrate potential uses of NATCS, including dialogue act classification and intent induction from conversations as potential applications, showing that dialogue act annotations in NATCS provide more effective training data for modeling real conversations compared to existing synthetic written datasets. We publicly release NATCS to facilitate research in natural dialog systems1.

heading: 1 Introduction
text: Applications that are applied to human-to-human customer support conversations have become increasingly popular in recent years. For example, assistive tools that aim to support human agents, provide analytics, and automate mundane tasks have become ubiquitous in industry applications (Amazon Contact Lens, 2023; Google Contact Center AI, 2023; Microsoft Digital Contact Center Platform,
1https://github.com/amazon-science/ dstc11-track2-intent-induction.
2023). Despite this growing interest in data processing for natural customer service conversations, to the best of our knowledge, there exist no public datasets to facilitate open research in this area.
Existing dialogue datasets focusing on development and evaluation of task-oriented dialogue (TOD) systems contain conversations that are representative of human-to-bot (H2B) conversations adhering to restricted domains and schemas (Budzianowski et al., 2018; Rastogi et al., 2020). Realistic live conversations are difficult to simulate due to the training required to convincingly play the role of an expert customer support agent in non-trivial domains (Chen et al., 2021). Existing datasets are also primarily written rather than spoken conversations as this modality is cheaper to simultaneously collect and annotate asynchronously through crowdsourcing platforms.
To address these gaps, we present NATCS, a multi-domain dataset containing English conversations that simulate natural, human-to-human (H2H), two-party customer service interactions. First, we describe a self-collection dataset, NATCSSELF, where we use a strict set of instructions asking participants to write both sides of a conversation as if it had been spoken. Second, we present a spoken dataset, NATCSSPOKE, in which pairs of participants were each given detailed instructions and asked to carry out and record conversations which were subsequently transcribed. We observe that the resulting conversations in NATCS share more characteristics with real customer service conversations than pre-existing dialogue datasets in terms of diversity and modeling difficulty.
We annotate a subset of the conversations in two ways: (1) we collect task-oriented dialogue act annotations, which label utterances that are important for moving the customer’s goal forward and (2) we categorize and label customer goals and goalrelated information with an open intent and slot
9652
schema, mimicking the process for building a TOD system based on natural conversations. We find that classifiers trained with the resulting dialogue act annotations have improvements in accuracy on real data as compared to models trained with preexisting TOD data.
Our main contributions are threefold:
• We present NATCS, a multi-domain dialogue dataset containing conversations that mimic spoken H2H customer service interactions, addressing a major gap in existing datasets.
• We show that NATCS is more representative of conversations from real customer support centers than pre-existing dialogue datasets by measuring multiple characteristics related to realism, diversity and modeling difficulty.
• We provide TOD dialogue act and intent/slot annotations on a subset of the conversations to facilitate evaluation and development of systems that aim to learn from real conversations (such as intent and slot induction), empirically demonstrating the efficacy of the dialogue annotations on real data.
Our paper is structured as follows: In Section 2, we review other dataset collection methods and approaches for evaluating dialogue quality. In Section 3, we describe how NATCS conversations and annotations were collected. In Section 4, we compare NATCS with pre-existing dialogue datasets as well as real H2H customer service conversations. Finally, in Section 5, we further motivate the dataset through two potential downstream applications, task-oriented dialogue act classification and intent induction evaluation.

heading: 2 Related Work
text: Dialogue Dataset Collection The goal of NATCS, to produce conversations that emulate real spoken customer service interactions, differs substantially from previous synthetic dialogue dataset collections. Previous synthetic, goal-oriented datasets have used the Wizard of Oz framework (Kelley, 1984). This framework calls for one person to interact with what they think is a computer, but is actually controlled by another person, thus encouraging a human-to-bot (H2b) style. Wen et al. (2016) define a specific version of this approach to be used with crowdsourced workers to produce synthetic, task-oriented datasets. This approach
has since been adopted as a standard method to collect goal-oriented conversations (Peskov et al., 2019; Byrne et al., 2019; Budzianowski et al., 2018; El Asri et al., 2017; Eric et al., 2017).
MultiDoGO (Peskov et al., 2019) (MDGO), SGD (Rastogi et al., 2020), and TaskMaster (Byrne et al., 2019) are particularly relevant comparisons to our work. MDGO explicitly encourages dialogue complexities, which serve as inspiration for our complexity-driven methodologies. SGD is interesting partially because of the scale of their collection, relevance of their task-oriented dialogue act and intent/slot annotations, and because they diverge from the common practice of using the Wizard-of-Oz methodology through the use of dialogue templates and paraphrasing. TaskMaster presents a methodology for self-collected dialogues, which is analogous to our NATCSSELF collection.
Despite these similarities, the methodology for NATCS differs significantly from any of these preexisting datasets, because of both the target modality (spoken, or spoken-like conversations), and the setting (H2H instead of H2B).
Analysis of Dialogue Quality An important component of our work is the comparison of synthetic dialogue datasets with real data. We adopt multiple metrics for comparing NATCS with both real and previous TOD datasets. Byrne et al. (2019) use perplexity and BLEU score as stand-ins for “naturalness”, with the logic that a dataset should be harder for a model to learn if it is more realistic, because realistic data tends to be more diverse than synthetic data. Previous collections of dialogue datasets also have made comparisons based on surface statistics such as the number of dialogues, turns, and unique tokens. Casanueva et al. (2022) compares intent classification datasets based on both lexical diversity metrics and a semantic diversity metric computed using a sentence embedding model.
There are a number of metrics more common in other sub-domains that may be useful for measuring naturalness in dialogues. The measure for textual lexical diversity (MTLD), discussed in depth by McCarthy and Jarvis (2010), provides a lexical diversity measure less biased by document length. Liao et al. (2017) introduce a dialog complexity metric, intended for analyzing real customer service conversations, which is computed by assigning importance measures to different terms, computing
utterance-level complexity, and weighting the contribution of utterances based on their dialogue act tags. Hewitt and Beaver (2020) performed a thorough comparison of the style of human-to-human vs. human-to-bot conversations, using lexical diversity measures, syntactic complexity, and other dimensions like gratitude, sentiment and amount of profanity, though the data was not released.

heading: 3 Methodology

heading: 3.1 Collection Methods
text: We propose two collection methods as part of NATCS. We have three goals for the resulting conversations: (1) They should exhibit the spoken modality, (2) all conversations from each domain should seem to be from the same company, and (3) they should appear to be real, human-to-human conversations between a customer and an agent. We explore two methodologies, resulting in the NATCSSPOKE and NATCSSELF datasets, to weigh collection cost and complexity compared to dataset effectiveness.
To support goal (3), we propose a set of discourse complexity types. The motivation for providing specific discourse complexities is to encourage some of the noise and non-linearity present in real human-to-human conversations. Based on manual inspection of 10 transcribed conversations from a single commercial call center dataset, we identify a combination of human expressions (social niceties, emotionally-charged utterances), phenomena mimicking imperfect, non-linear thought processes (change of mind, forgetfulness/unknown terminology, unplanned conversational flows), re-
flections of the wider context surrounding the conversation (continuing from a previous conversation, pausing to find information), distinctions between speakers’ knowledge bases, and the use of multiple requests in single utterances (stating multiple intents, providing multiple slot values). A list of these complexities along with estimated target percentages (minimum percent of conversations where these phenomena should be present) is provided in Figure 1. Descriptions and examples are provided in Appendix Table 15.
To achieve cross-dataset consistency, supporting goal (2), collectors are provided with mock company profiles, including name of the mock company, as well as mock product or service names with associated prices. Collectors are also provided with a schema of intents and associated slots. Some
flexibility is allowed in the slot schema to reflect real world situations where customers may not have all requested information on hand. Examples of company profiles and intent schemas are provided in Appendix Tables 13 and 14. For each conversation, we sample a set of minimum discourse complexity types. For example, one conversation could be assigned the target complexities of ChitChat, FollowUpQuestion, MultiElicit, and SlotLookup. Scenarios eliciting each of these complexity types are generated and provided to the participants.
For NATCSSPOKE, one participant plays the part of the customer service representative (“agent”), and one participant plays the part of the customer (“customer”). The participants are recorded as they play-act the scenarios described on their instruction sheets from the same room. These audio recordings are then transcribed and annotated for actual complexities.
Given the time, cost, and complexity involved for the creation of the NATCSSPOKE datasets, as an alternative approach, we apply the NATCSSELF method. For NATCSSELF, participants write selfdialogues as if they were spoken out loud. This method has the benefit of (1) not needing to be transcribed, and (2) requiring only one participant to create each conversation and therefore not requiring scheduling to match participants together. However, these rely on an understanding by the participants of the distinction between spoken and written modality data. The NATCSSELF method follows a similar set-up as the NATCSSPOKE method, except in addition to being provided with a set of target discourse complexities, participants are also provided with a set of spoken form complexities. While discourse complexities target discourserelated phenomena, spoken form complexities consist of phenomena specifically observed in spoken form speech. For this complexity type, we include phenomena such as hesitations or fillers (‘um’), rambling, spelling, and backchanneling (‘uh huh go on’). A list of these complexities along with target percentages is provided in Figure 4, and further examples are provided in Appendix Table 16.

heading: 3.2 Annotations
text: One goal of collecting realistic dialogues is to facilitate the development and evaluation of tools for building task-oriented dialogue systems from H2H conversations. To this end, we perform two types of annotations on a subset of NATCS: Dialogue
Act (DA) annotations and Intent Classification and Slot Labeling (IC/SL) annotations.
IC/SL annotations are intended to label intents and slots, two key elements of many TOD systems. An intent is broadly a customer goal, and a slot is a smaller piece of information related to that goal. We use an open labelset, asking annotators to come up with specific labels for each intent and slot, such as “BookFlight” and “PreferredAirline” as opposed to simply “Intent” and “Slot”. Annotators are instructed to label the same intent no more than once per conversation. For slots, we use the principle of labeling the smallest complete grammatical constituent that communicates the necessary information.
Our DA annotations are intended to identify utterances that move the dialog towards the customer’s goal. TOD systems often support only a small set of dialogue acts that capture supported user and agent actions. For the agent, these may include eliciting the user’s intent or asking for slot values associated with that intent (ElicitIntent and ElicitSlot respectively). For the user, such acts may include informing the agent of a new intent or providing relevant details for resolving their request (InformIntent and InformSlot respectively). Such acts provide a limited view of the actions taken by speakers in natural conversations, but do provide a way to identify and categorize automatable interactions in natural conversations. Table 1 provides an example conversation annotated with intents, slots, and dialogue acts.

heading: 4 Dataset Analysis
text: To better motivate NATCS as a proxy for natural, spoken form customer service conversations from multiple domains with a diverse set of intents, we compare with real conversations from commercial datasets comprising 5 call centers for retail and finance-related businesses (henceforth REAL). All datasets in REAL consist of manuallytranscribed conversations between human agents and customers in live phone conversations where all personally-identifiable information has been preredacted. We restrict our analysis to datasets with primarily customer-initiated two-party dialogues.
As shown in Table 2, one surface-level distinction from publicly available TOD datasets is the average number of turns per conversation. Compared to MDGO, SGD, MWOZ and TM1Self, REAL has considerably longer conversations (over 70 turns
per conversation on average, vs. 22 for TM1Self). Furthermore, each turn has more words per turn, suggesting increasing complexity in spoken H2H dialogues. NATCS closely matches REAL in terms of conversation and turn lengths.

heading: 4.1 Intents and Slots
text: Table 3 provides a comparison of intent and slot annotations between existing synthetic datasets, NATCS and REAL. Datasets in REAL contain considerably more intents and slots for a particular domain than existing TOD datasets like MDGO. Turns containing intents are longer for both NATCS and REAL than SGD and MDGO.
Figure 2 compares the intent and slot distributions between RetailA and NATCSSELF Insurance, indicating that both have skewed, long-tailed distributions of intents/slots, a product of the open intent/slot schemas used in NATCS.

heading: 4.2 Diversity Metrics
text: As we expect conversations in REAL to be less homogeneous than synthetic dialogue datasets, we compute automatic metrics to measure multiple aspects of diversity and compare with NATCS.
Conversational Diversity In Table 3, we examine the diversity of conversation flows as measured by the ratio of unique sequences of slots informed
by the customer to the total number of sequences (e.g. slot bigrams or trigrams). In SGD, which constructs dialogue templates using a simulator, we observe a much lower percentage of unique n-grams than in REAL, despite SGD containing dialogues spanning multiple domains and services. On the other hand, while NATCS has lower slot n-gram diversity than REAL, both collection types have substantially higher slot n-gram diversity scores than both MDGO or SGD.
The average perplexity of a language model provides one indication of the difficulty of modeling
the dialogue in a given dataset (Byrne et al., 2019). High perplexity indicates higher difficulty, while low perplexity can indicate more uniform or predictable datasets. We compare between fine-tuning and zero-shot language modeling settings using GPT-NEO (Black et al., 2021). Zero-shot evaluation gives an indication of how compatible the dataset is with the pre-trained model without any fine-tuning. Section A.1 provides details on the fine-tuning procedure.
As shown in Table 4, we observe high perplexity on real data and low perplexity on synthetic datasets like SGD and MDGO. NATCS has lower perplexity than REAL, but considerably higher perplexity than MDGO, MWOZ, and SGD. Interestingly, there is a wider range of perplexities across real datasets, while most existing TOD datasets have a perplexity of 10 or less.
Intent Diversity We also investigate the semantic diversity of intent turns (SemDivintent) following Casanueva et al. (2022). Details of this cal-
culation are provided in Section A.1. As shown in Table 3, we observe the highest SemDivintent for REAL, but NATCS has considerably higher SemDivintent than pre-existing synthetic datasets, indicating greater potential modeling challenges. We also compare the semantic of diversity of NATCS with other datasets for specific aligned intents, like CheckBalance in Appendix Table 9, also observing higher semantic diversity as compared to pre-existing intent classification benchmarks. Further investigation into the lexical diversity of intents is provided in Section A.1.
NATCS Dialogue Acts Distribution To better understand the characteristics of typical H2H customer service dialogues in call centers, we annotate a subset of real conversations with the “taskoriented” dialogue acts described in Section 3.2. Because NATCS dialogue acts consist of a small set of intent and slot-related functions commonly employed in automated TOD systems, we do not expect the labels to have high coverage in real conversations that do not revolve around a fixed set of intents and slots. However, they aim to provide a mechanism for aligning turns from natural conversations onto these automatable TOD constructs.
Table 5 compares the percentage of turns labeled with NATCS dialogue acts across multiple datasets along with the average counts of each label per dialogue. As expected, compared to synthetic data, dialogues in REAL have fewer turns labeled with NATCS dialogue acts (27.7%). More reflective of REAL in this regard, in NATCSSPOKE and NATCSSELF, we observe that more than half of the turns in each conversation are not labeled, despite having higher total counts of dialogue acts per con-
versation as compared to pre-existing datasets.

heading: 4.3 Human Evaluation
text: We also perform a human evaluation comparing MDGO, SGD, and REAL datasets against both NATCSSELF and NATCSSPOKE. Rather than compare complete dialogues, because of the large disparity in conversation lengths, we restrict evaluation only to snippets including the first 5 turns after an intent is stated (including the turn containing the intent). Conversation snippets are graded on a scale of 1 to 5 along multiple dimensions, including realism (believability of the dialogue), concision (conciseness of customer, lack of verbosity), and spoken-likeness (possibility of being part of a spoken conversation). See Section A.2 Figure 3 for explicit definitions provided to graders.
The evaluation is conducted by 6 dialogue systems researchers, with each grader rating 50 randomly-selected conversations. As indicated by results in Table 6, conversations from REAL observe high values for both realism and spokenlikeness (4.71 and 4.95 respectively), with lower values for concision, indicating greater customer verbosity in real dialogues. The results also indicate that although expert human graders can still differentiate NATCS from real conversations, NATCS is graded as significantly more realistic and indicative of spoken modality than SGD and MDGO (two-tailed T-test with p < 0.005).

heading: 5 Applications
text: In this section, we investigate two potential applications of NATCS as a resource for building and evaluating systems related to human-to-human (H2H) customer service interactions. One goal of NATCS is to encourage research in the under-

heading: Dataset Realism Concision Spokenlike
text: explored space of H2H task-oriented interactions, so these are intended to serve as motivating examples rather than prescribed uses.

heading: 5.1 NATCS as Training Data
text: One goal of this work is to accelerate the development of dialogue systems based on H2H conversations. While most existing work in intent induction assumes that customer turns corresponding to requests have already been identified, NATCS dialogue acts provide a mechanism to map turns onto TOD constructs like intents.
To validate the usefulness of dialogue act annotations in NATCS, we compare the cross-dataset generalization of dialogue act classifiers trained on annotations in NATCS against that of SGD, a large multi-domain corpus of task-oriented dialogues, evaluating on real conversations between human agents and customers.
We fine-tune ROBERTA-BASE using per-label binary cross entropy losses to support multiple la-
bels per sentence. Fine-tuning details are provided in Section A.3. We compare the dialogue act classification performance on real data when training on SGD, NATCS, and in-domain vs. out-of-domain real data. As shown in Table 7, a DA classifier trained on NATCS performs significantly better on real data than a classifier trained on SGD. Performance still lags behind that of training on real data, but with NATCS, the gap is closed considerably. In Section A.4, we also show that the dialogue act annotations in NATCS are able to generalize to new domains.

heading: 5.2 Intent Clustering with Noise
text: Recent work indicates growing interest in applications that can accelerate the development of these systems by automatically inducing TOD constructs such as intents and slots from customer support interactions (Yu et al., 2022; Shen et al., 2021; Kumar et al., 2022; Perkins and Yang, 2019; Chatterjee and Sengupta, 2020). To further motivate NATCS as a realistic test bed for applications that learn from natural conversations, we demonstrate how it can serve as a benchmark for unsupervised intent clustering tasks.
In a realistic setting, turns in conversations containing intents will not be provided in advance. We thus compare three settings: 1) using the first customer turn in each conversation 2) using turns predicted as having intents with a dialogue act classifier and 3) using turns labeled with intents (gold dialogue acts).
Utterances are encoded using a sentence embedding model from the SENTENCETRANSFORMERS library (Reimers and Gurevych, 2019), ALLMPNET-BASE-V2. We use k-means clustering with the number of clusters set to the number of reference intents. To assign cluster labels to all gold input turns, we use label propagation by training

heading: Turns NMI ACC Purity Inv. Purity
text: a logistic regression classifier using ALL-MPNETBASE-V2 embeddings as static features on inputs assigned cluster labels (such as first turns), then apply the classifier to the missing turns to get predicted cluster labels.
The results, shown in Table 8, demonstrate that using automatically predicted turns leads to a drop in purity and NMI. The drop in purity is attributable to irrelevant, non-intentful turns being clustered together with relevant intents, a potentially costly error in real-world settings that is not typically reflected in intent clustering evaluation.

heading: 6 Conclusions
text: We present NATCS, a corpus of realistic spoken human-to-human customer service conversations. The collection of NATCS is complexity-driven and domain-restricted, resulting in a dataset that better approximates real conversations than preexisting task-oriented dialogue datasets along a variety of both automated and human-rated metrics. We demonstrate two potential downstream applications of NATCS, showing that training using NATCS results in better performance with real test data compared to training using other publiclyavailable goal-oriented datasets, and that NATCS can provide a new challenging benchmark for realistic evaluation of intent induction.
We hope that NATCS will help facilitate open research in applications based on customer support conversations previously accessible mainly in industry settings by providing a more realistic annotated dataset. In future work, we hope to expand on annotations in NATCS to support more tasks such as call summarization, response selection, and generation.

heading: Limitations
text: NATCS is partially annotated with dialogue acts, intents, and slots, which are annotated independently from the initial collection of the conversations. While decoupling annotations from collection was intended to facilitate natural and diverse dialogues, the methodology is more timeconsuming and expensive than previous approaches that use pre-structured conversation templates to avoid the need for manual annotation. In particular, NATCSSPOKE requires multiple participants engaging in synchronous conversations, followed by independent manual transcriptions and annotations, making the approach particularly time-consuming and difficult to apply for large collections. Furthermore, this decoupling of annotations from collection has greater potential for annotator disagreement.
While the complexity types and annotations are mostly language-agnostic, NATCS is restricted to EN-US customer-initiated customer service conversations between a single agent and customer in a limited number of domains (multi-party conversations beyond two participants or agent-initiated conversations are not included). The annotations included are primarily intended for applications related to task-oriented dialogue systems.
Further, we note that NATCS closes the gap from real conversations along many metrics, but still falls short along some dimensions. We find that real conversations are more verbose, more believable, and less predictable. We also note that comparisons in our paper focused on a limited number of taskoriented dialogue datasets with different collection approaches, and did not exhaustively include all pre-existing dialogue datasets for comparison.

heading: Ethics Statement
text: In this paper, we present a new partially-annotated dataset. In adherence with the ACL code of conduct and recommendations laid out in Bender and Friedman (2018) it is appropriate to include a data statement. Our dataset is completely novel, and was collected specifically to support the development of natural language systems. Workers who are proficient in the EN-US variant of English were hired through a vendor with a competitive hourly rate compared to the industry standard for language consultants. For NATCSSPOKE, these workers spoke to each other and then transcribed the data. For NATCSSELF these workers wrote the conversations.
To annotate the data, we used two pools of annotators. Both had formal training in linguistics and were proficient in the EN-US variant of English. One pool was hired through a vendor with a competitive hourly rate. The other pool consisted of full-time employees.
Curation Rationale Our dataset includes all of the data that was produced by the consultants we hired. Quality Assurance was done on a subset of this data. We hope that any concerns would have shown up in this sample. We annotated a random subset of the full dataset.
Language Variety The dataset is EN-US. The speakers (or writers) were all fluent speakers of EN-US. We did not target a particular sub-type of the EN-US language variety.
Speaker Demographics We do not have detailed speaker demographics, however, we do have male and female speakers from a variety of age ranges.
Annotator Demographics We do not have detailed annotator demographics, however, we do have male and female speakers from a variety of age ranges. All annotators had at least some formal linguistics training (ranging from a B.A. to a Ph.D.).
Speech Situation For NATCSSPOKE, speakers were talking in real time on the phone to one another. It was semi-scripted. Speakers were not told exactly what to say, but were given some constraints.

heading: A Appendix

heading: A.1 Experimental Setting Details
text: In this section, we provide details on the experimental settings used for evaluation of NATCS and other dialogue datasets.
Perplexity Evaluation To evaluate the language modeling (LM) perplexity on each dataset, we compare both a fine-tuning setting as well as a zero-shot setting using GPT-NEO (Black et al., 2021) as the
pre-trained LM. We fine-tune GPT-NEO on each dataset, sampling 4096 blocks of 128 tokens as training data and evaluating on held-out test splits. Fine-tuning is performed for 6 epochs with a batch size of 64 and learning rate of 5e-5. Perplexity is computed at the level of bytes using a sliding window of 128 tokens.
Semantic Diversity Evaluation Following Casanueva et al. (2022), to compute semantic diversity for a single intent, we (1) compute intent centroids as the average of embeddings for the turns labeled with the intent using the SentenceBERT (Reimers and Gurevych, 2019) library with the pre-trained model ALL-MPNET-BASE-V2, then (2) find the average cosine distance between each individual turn and the resulting centroid. Finally, (3) overall semantic diversity scores (SemDivintent) in Table 3 are computed as a frequency-weighted average over intent-level scores.
Table 9 shows the semantic diversity scores (Casanueva et al., 2022) for different intents aligned across several datasets.
Lexical Diversity of Intents As an additional indicator of intent-level diversity, we measure the frequency-weighted average of type token ratios for utterances within each intent (TTRintent). To account for the redaction of names and numbers in real data, we perform a similar redaction step on all datasets, automatically converting names and numbers to a single PII token with regular expressions and a named entity tagger before computing TTR. Shown in Table 10, we observe similar TTRintent between NATCS and REAL, while the pre-existing synthetic datasets lag behind considerably.

heading: A.2 Human Evaluation Details
text: Guidelines provided to human graders are provided in Figure 3. Realism measures the overall believability that the conversation could have taken place in a real scenario (which penalizes unlikely, silly utterances from the customer or unprofessional behavior from the agent). Concision measures how concise the customer responses are, with lower scores for lengthy utterances containing details that are unnecessary for resolving their request. Spoken-like measures the likelihood that the conversation was originally spoken in a phone conversation (as opposed to being written from a chatroom or messaging platform).
50 of the 300 conversation snippets evaluated were graded by pairs of annotators (5 pairs total).
For these, we observed a Krippendorff’s alpha of 0.52, 0.59, and 0.37 respectively for Spoken-like, Realism, and Concision respectively.

heading: A.3 Dialogue Act Classifier Training Details
text: The dialogue act classifier is implemented by finetuning ROBERTA-BASE (Liu et al., 2019) using per-label binary cross entropy losses to support multiple labels per sentence. To encode dialogue context, we append the three previous sentences to the current turn with a separator token ([SEP ]), adding a speaker label to each sentence and using padding tokens at the beginning of the conversation. Fine-tuning is performed for 6 epochs with a batch size of 16, using AdamW with a learning rate of 2e5. Dataset-level dialogue act classifier performance is provided in Table 11.

heading: A.4 Dialogue Act Cross-Domain Generalization
text: For models trained on NATCS to be useful in practice, they must be able to generalize beyond the limited number of domains present in NATCS. To measure cross-domain generalization, we perform cross validation, training separate models in which we hold out a single domain (e.g. train on Banking, evaluate on Travel). As shown in Table 12, performance on the heldout domains is lower, particularly
for recall, but does not drop substantially.

heading: A.5 Collection Methodology
text: Table 13 provides an example company profile used to assist in achieving cross-dataset consistency. Table 14 provides an example intent/slot schema for a ResetPassword intent in the Insurance domain. Table 15 provides example utterances for each discourse complexity. Table 16 provides examples of spoken complexities used to simulate spoken-form conversations in the Insurance.
Figure 4 provides target percentages for complexities used to simulate spoken conversations in NATCSSELF.

heading: A.6 Conversation Examples
text: Tables 17, 18, and 19 provide examples of conversations collected in NATCSSPOKE. Tables 20, 21, and 22 provide examples of conversations collected in NATCSSELF.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
3 A2. Did you discuss any potential risks of your work? 7
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3,4,5
3 B1. Did you cite the creators of artifacts you used? 2,3,4,5
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? License information is embedded in the released datasets.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The artifacts used (models) have standard licenses and expectations of use. The created artifacts are not derivative, and usage restrictions are embedded in the release license.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 8 (Ethics Statement) and 4 (Dataset Analysis), discussing PII redaction.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 8 (Ethics Statement)
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4 (Dataset Analysis)
C 3 Did you run computational experiments? 4 (Dataset Analysis), 5 (Applications), Appendix
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? The paper does not focus on modeling improvements, but instead of analysis of data – these details were deemed unnecessary.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4 (Dataset Analysis), 5 (Applications)
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4 (Dataset Analysis), 5 (Applications), Appendix
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3 (Methodology), 4 (Dataset Analysis), 8 (Ethics Statement)
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Partial instructions were provided (Appendix), however, full details cannot be released for legal reasons.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 8 (Ethics Statement)
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? 8 (Ethics Statement)
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not requires for this kind of data collection.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? 8 (Ethics Statement)

heading: Future Work
text: In future work, we hope to expand on annotations in NATCS to support more tasks such as call summarization, response selection, and generation