Does Collaborative Human–LM Dialogue Generation Help Information Extraction from Human–Human Dialogues?
Authors: Bo-Ru Lu
Nikita Haduong
Chia-Hsuan Lee
Zeqiu Wu
Hao Cheng
Paul Koester
Jean Utke
Tao Yu
Noah A. Smith
Mari Ostendorf
Venue: COLM (2024)
Keywords: Data Synthesis, Data Annotation, Human–LM Collaboration
TL;DR: The study introduces a human-in-the-loop dialogue generation framework that significantly improves task performance by synthesizing realistic call center dialogues, demonstrating the value of human-LM collaboration in data generation and annotation.

Abstract: The capabilities of pretrained language models (LMs) have opened opportunities to explore new application areas, but applications involving human-human interaction are limited by the fact that most data is protected from public release for privacy reasons. Problem-solving human-human dialogues in real applications can be much more complex than existing Wizard-of-Oz collections, preventing successful domain transfer. To support information extraction (IE) for a private call center dataset (AIC), we introduce a human-in-the-loop dialogue generation framework capable of synthesizing realistic dialogues. In IE experiments with AIC dialogues, we observe 25% relative improvement in F1 after augmenting a small set of real human-human conversations with synthetic data. In controlled experiments, we compare training with our human-in-the-loop-synthesized data vs. fully automatically LM-generated data and find that collaborating humans adds value both in the generation and annotation stages. We release code and our synthetic dataset to illustrate the complexity of call center conversations and encourage development of complex dialogue datasets that are more representative of natural data.

Published as a conference paper at COLM 2024
Does Collaborative Human-LM Dialogue Generation Help Information Extraction from Human-Human Dialogues?
Bo-Ru Lu
Nikita Haduong
Chia-Hsuan Lee Zeqiu Wu
Hao Cheng
Paul Koester
Jean Utke
Tao Yu + Noah A. Smith Mari Ostendorf
University of Washington
" Microsoft Research Allstate
University of Hong Kong
à Allen Institute for Al
roylu@washington.cm edu quêcs. weeeesswwwiinnton..om edu
Abstract
The capabilities of pretrained language models (LMs) have opened op- portunities to explore new application areas, but applications involving human-human interaction are limited by the fact that most data is protected from public release for privacy reasons. Problem-solving human-human dialogues in real applications can be much more complex than existing Wizard-of-O2 collections, preventing successful domain transfer. To sup- port information extraction (IE) for a private call center dataset (AIC), we introduce suman-in--he-loos dialogue generation framework capable of synthesizing realistic dialogues. In IE experiments with AIC dialogues, we observe 25% relative improvement in Fi after augmenting small set of real human-human conversations with synthetic data. In controlled experi- ments, we compare training with our -aama--intthelloo--yntteeeeee vs. fully automatically L.M-genrrate data and find that collaborating hu- mans adds value both in the generation and annotation stages. We release code and our synthetic dataset to illustrate the complexity of call center conversations and encourage development of complex dialogue datasets that are more representative of natural data.
data
1 Introduction
Rapid advances in natural language processing have driven interest in its use in wide variety of domains. However, applications that involve human-human interaction, such as call center dialogues, have had limited success (Lam et al. 2019 Albrecht et al. 2002 Pezik et al. | 2022: Lu al. 2022). One reason is that natural problem problem-sollvigg dialogues are not typically publicly available for privacy reasons, restricting opportunities for researchers to explore methods in advancing applications for these domains. Further, annotating private datasets can be expensive because of the need for in-house expertise, training resources are limited. In this paper, we introduce a method to fill the data gap using synthetic data generated by collaborative haman-language model framework. Specifically, we experiment with a task of extracting information from auto insurance call center dialogues, using public synthetic data to improve performance on private dataset.
Many available dialogue datasets are designed for training virtual agents, collected using pairs of humans to perform task (Budzianowski et al., 2018 Rastogi et al.) 2020 Chen et al. 2021). Designing for human-machine interaction results in dialogues that lack the complexity of human-huma
dialogues. Additionally, human-only data collection can have limited content diversity, result in imbalanced training sets, and does not scale to more complex tasks, due to the high cost of employing domain experts (Gururangan et al. 2018 Geva et al. 2019 Qian et al. 2021).
To reduce data collection costs, researchers have explored the use of language models (LMs)
to generate synthetic training data (Liu et al. 2022a Wang et al. 2023 Bao et al. 2023 Li
2 "Equl contribution. Code & data are available at httpp: htttttps://boourrollll thtb....oo So/DialGen!

---

Published as a conference paper at COLM 2024
Global
Can you provide me with more details about the accident?
Iavolved Cars
Location
Highway
Yes, picked up passenger that asked to take him to Santa Monica Pier. I was taking the 1-10 and I pulled over onto the shoulder to check on the enginn because thought heard rattling noise. I couldn't find anything. but then
Caller
Uber/Latt
Yes Santa Monica Pier
Destination of Trip
got in the and preparing to start cirivin again, Subaru BRZ suddenly collided with me.
Purpose of Trip Car Motion
I picked up Stopped
PPassengess
Other Deiver
Males/Model
Subaru BRZ
Can you tell me more about the damages to both vehicles?
Caller
Yes, my car suffered significant damage to the front bumper, hood, and headlights. The had damage to left and front fender.
Damage Part
Other Driver
Make Model
Subara BRZ
Damage Part
Left, Front
Sarah, can you remind how many passengers the with you the time of the accident?
Caller
Cops, sorry about that. I misrermembere.. There were actually two passengers in the car with me.
Passengers
Client
Agent
Figure 1:
An illustrative snippet of our dialogue with entity-slot-value triples. Yellow is the slot with multiple values. Italic blue and yellow are the same slot (Damage Part) with different entities (e.g,, Caller and Other Driver). Red is a slot with a value update.
et al. 2023 Asai et al.
2024). Synthesized data can target long-tail phenomena (Chu et al. 2020: Zhao et al. 2023 Zhang et al. 2023 Henning et al.) 2023) and allow for public release
of data that closely em unnulates real-world privacy-constrained domains, such as the medical domain (Park et al. 2018). Although LMs can follow instructions to generate text that closely resembles human writing, there can be challenges to ensure that the data are diverse and not too simplistic (Stahlberg & Kumar 2021 Liu al. 2022a). In addition, they still suffer from incoherence and consistency issues (Clark et al. 2021 Dou et al. 2022). To mitigate the shortcomings of LMs, human-LM collaboration can offer a robust solution, leveraging the
strengths of both humans and machines (Sharma et al. 2022 Uchendu et al. 2023).
The idea of integrating human intelligence with artificial intelligence was initially intro- duced in Licklider (1960). Recent research has highlighted the proficiency of human-LM collaboration generating a variety of data; however, most of the study focuses on short dialogues and text (Liu et al., 2022a; Bonaldi et al. 2022). In contrast, we investigate using human-in-the- tn-o framework to create lengthy and complex dialogues.
Our work proposes human-LM collaborative framework for dialogue generation (DIAlGEN) that leverages the scalability and creativity of generative models, yet retains controllability through humans. Human collaborators edit the synthesized dialogues, which we use to boost information extraction performance on real-world call center data.
Many call center dialogues involve problem solving where customers provide information to an agent through question-answer pairs and clarifications that need to be interpreted in the context of the dialogue history. Our information extraction (IE) task thus framed as an iterative information update after each agent-customer exchange, analogous to dialogue state tracking (DST) in task-oriented dialogues. However, unlike DST, the information extracted from each turn is collected to create a summary of the call rather than to generate a virtual agent's response or make an API call. In addition, the summary includes entities that are associated with attributes (slots) and values. To evaluate models on this IE task, we introd uce entity-centric scoring methods that allow partial matching of multiple and descriptive values.
We demonstrate the effectiveness DIALGEN by generating data in auto insurance calls, a domain with privacy restrictions that prevent public release of actual call recordings, and by performing information extraction. We work with a private dataset containing 34 dialogues with an average 197 utterances per dialogue and synthesize 235 dialogues with an average 46 utterances per dialogue. Experiments in our IE task show that additional synthetic data relatively improves model performance by 25% in the full F score.
2

---

Published as a conference paper at COLM 2024
To summarize, main contributions are:
We design DIALGEN, a collaborative human-LM framework for generating complex dialogues in domains where privacy constraints have previously prevented data sharing with the research community. Synthetic data, training documentation and prompts will be released.
We present DIALGEN-AIC, a custom dataset designed to illustrate the complexity of real-world auto insurance call center data. While not intended as benchmark, DIALGEN-AIC aims to provide demonstration of the complex nature of real con- versations and the challenges faced in this domain, including linking information with different entities and tracking multiple values in single slot.
We propose an entity-centric scoring methodology that considers information links to different entities, allows for multiple slot values, and provides partial match scores for descriptive values.
We compare our DIALGEN framework against fully automatic LM framework and find that human collaboration adds value during both in generation and annotation.
Prompt Creation
Subdialogue Generation
Full Dialogue
Apest: LtI. shank you farelllng!!
Apant: What the make/model your car
Cullan
Abrl
Vakee
Ciller: 1h's owhnn
Taok therripttin
make/msoodd
Tripkts
oCyou car?
Cullan DS's
Tulk
Story
Regenerate
Hoods Accved.
Personalties
Agest: Ainghe then, take cane, Andrew, and lel know if noed any tutther
Paloge Hastery
What is the maloy/motet of your soden 20u5 Hooda Caler.
Agest
Revise
Thank you,
Figure 2: In the DIALGEN framework, a LM and human reviewer collaborate to generate dialogue. The LM creates subdialogues based on a scenario and dialogue history. The reviewer evaluates the subdialogue and can have the LM regenerate the subdialogue before revising it. The revised subdialogue is added to the dialogue history for generating the next subdialogue. This iterative process continues until the dialogue complete.
Dialogue generation (DIALGEN)
As shown in Figure 2 our DIALGEN framework is designed generate schema-guided dialogues through human-LM collaboration. An LM is selected as the backbone, then the data generation process begins with an initial task prompt consisting of natural language description for the desired dialogue (e.g., task description, desired slots, story, and per- sonalities) and dialogue history. During each iteration, the LM first proposes candidate subdialogue based on the history (the initial task prompt and the generated conversation so far). A human reviewer with sufficient domain knowledge then validate, edit and annotate the subdialogue, before requesting a continuation via updated prompt to the LM. The reviewer can optionally augment the prompt with a specific instruction (see Table in Appendix D) related to the desired dialogue flow. This process repeats until the dialogue is mechanism ensures that the resulting dia-
complete. At a high level, the human-in-the-lond logues are coherent and consistent with the prompt, covering desired content and fulfilling style specifications from domain experts. In the following, we describe each component of DIALGEN in detail.
2.1 Prompt for dialogue generation
The prompt for generating synthetic dialogues includes: the task description, entity-slot- value triplets, story, personality and dialogue history
An example of full prompt is given in Appendix D.1
3

---

Published as a conference paper at COLM 2024
Task Description. Similar to the task descriptions given to humans in Wizard-of-Oz setups (Kelley) 1984), the template-based task description gives the information about the dialogue participants and the task scenario for the conversation, such as having the LM role-play user calling to file a claim with an agent at an insurance company, e.g,, "Role play car accident claim call. One person is Alice, agent for car insurance company, and the other is Bob, the caller who wants to file a claim."
Entity-slot-value Triplets. We randomly sample entity-slot-value triples from the expert- authored ontology to steer the LM to generate required content in the dialogue, allowing precise coverage of specific information, e.g., (Caller, Injury, Neck).
Story. Kim et al. (022) synthesize social dialogues from triples of common sense knowledge by first using a social narrative to set up the scenario. We similarly use the randomly sampled triplets to generate a story with the LM before the dialogue generation. For example, the aforementioned entity-slot-value triple will be converted into the snippet of a story: "The impact of the collision caused Bob's car to spin around and come to a stop. He immeddately felt a sharp pain in his neck and knew that something was wrong."
Personality. To enrich the diversity of callers, we randomly sample personality from the list (Table 9n A Appendix[D for each dialogue, e-g., "Bob is feeling distressed or frustrated due to the accident and its consequences." For the agent, we use the same personality for all dialogues, e.g., "Alice is conversational, personable, putient, empathetic, sympathetic and professional."
Dialogue History. The LM uses the full dialogue history to generate subdialogue that is consistent with the flow of the conversation. During the subdialogue generation, we append completed subdialogues before generating the next subdialogue. The initial dialogue history is always one exchange, e.g, "Alice: Hi, thank you for calling DialGen Insurance! This is Alice. How may help you today?" followed by "Bob: am calling regarding car accident." "
2.2 Human-in-the-loop subdialogue generation
The dialogue is generated iteratively where each subdialogge is revised by a human reviewer. Subdialogues are individually revised by a human trained to correct common LM errors such as those described by Dou et al. (2021), verify that required information is present (the sampled triples), and edit the text to meet stylistic criteria (e.g., adjusting tone). The reviewer can either revise individual turns directly or instruct the LM to regenerate specified turns, e.g., "Have the caller correct earlier incorrect information" (more examples in TTale 8 in Appendix DD. The LM may try to end the dialogue by including termination signals such as good bye." If the LM ends the dialogue without covering the required triplets, the reviewer can delete and regenerate the turns.
2.3 Dialogue annotation
After subdialogue is generated, human annotator are asked to label spans in the dialogue that have information tuples associated with the task ontology. If a tuple in turn f has a slot with the same referent and a different value than a previous turn, the human annotators are asked to resolve the duplication by indicating whether the new value is a correction UPDATE, KEEP, additional detail to be concatenated with the previous value CONCAT. This annotation step is optional and can be decoupled from the framework depending on the target tasks or domains.
3 Problem definition and evaluation
An auto insurance call center dialogue involves a customer working together with an agent to address an issue or submit claim. As the conversation progresses, the extracted information must be iteratively updated. This updating process is similar to the concept of dialogue state tracking (DST) used in task-oriented dialogues. However, unlike standard DST, the extracted information is used to summarize the call, not to make API calls or generate responses by virtual agent.
4

---

Published as a conference paper at COLM 2024
3.1 Problem definition
Extracted structured information is typically represented as a collection of tuples vs, e S}, where is slot label, is the associated value, and S is the full set of slots in the ontology. Values can be associated with a slot-dependent restricted set V or free-form text (e.g,, a home address) or null. For multi-domain systems where different domains share some but not all slots (e.g., many domains have date slot), the domain d is separately tracked: {dd, v), E D,s E S}. The full set of tuples updated after each agent-user exchange to support construction of application calls needed to complete the task.
We formalize our information extraction task as follows. Ignoring domain for brevity, define (A, U); as the pair of agent and user turns in exchange t. Given sequence of exchanges between an agent and a user, { {A, L),,,.,, (A, L)r find the dialogue state {(s, v), E } where S, is the subset of slots active at time t (i.e., having non-null values). The state associated with the final turn T effectively provides a summary of the information extracted
from the user in the dialogue.
3.2 Definition of extracted information
To accommodate the complexities of our dialogues, we augment the DST problem in three ways. First, we introduce the notion of "referent", either with the global context or with the entity with which the extracted information is associated. Second, we allow slots to take on multiple values. Lastly, we allow slot values to be updated in multiple ways: a value can be corrected by the user, a new value can be added to form list, or an existing value can be augmented, e.g,, with details expanding on a free-form slot. Figure 1 provides an example of an agent gathering information about an accident together with the extracted tuples. There are three referents (Global context, Caller, and Other Driver); the number of passengers in the caller's vehicle was corrected from one to two; and the other driver's 's car has ha multiple Damage Parts (left and front).
With these changes, we describe our notation as follows, using the arrow diacritic to indicate cumulative state elements, upper case to indicate tuples and lower case to indicate labels or values, boldface to indicate a set of tuples, and calligraphic font indicate set of values.
The initial dialogue state Xo is empty. The cumulative belief (CB) state X; (for t > 0) could
be predicted directly or via a recursive state update: X, = update( X (1)1,,;), where only new / updated state values are predicted in the turn-level belief (TLB) X; and the update function adds new slots and replaces updated slots. In the direct approach, it is possible to correct errors made by the model in previous turns, as well as introduce errors. A potential advantage of the update approach is that TLBs are shorter and therefore easier to predict.
Formally, X, and X, are defined as follows. Define R, as the set of referents mentioned in dialogue up through turn and R; C R, as the subset of referents associated with information updates in turn { The dialogue state and TLB after turn X, and Xt, respectively, can both be represented as a set of referent-assoocitted sets of active slots:
5 TT)r E R.) and X, ((r,S,), R2} where S , Sy,,, } frr is the number of active slots for referent updated at turn f, and S denotes the cunulative set of slots. An active slot is defined as S,j = (syj, Vrj), where Srj E S is the jth slot linked to the referent r, S the set of slot (or domain-slot) types, and Vrj is a set of one or more values (categorical or free form text) associated with that slot. For our generated data, annotators are asked to provide state updates.
3.3 Evaluation
In IE tasks, precision, recall, and the F-measure are commonly used, while DST is based on joint goal accuracy (JGA) and slot accuracy.
?urr application uses finite set of types R, S R, but it could be an open set, e.g. based on names.
5

---

Published as a conference paper at COLM 2024
Similar to DST, our IE task updates extracted information across turns. However, directly adopting DST metrics for dialogue-based IE is not ideal for two reasons. First, JGA is useful for DST because DST tasks require database queries that are built from the detected slots and values. Hence, accurate prediction is needed for all domains and slots, including null-valued instances. For a complex ontology, where many slots will be unfilled, JGA effectively emphasizes precision over recall. In contrast, for an IE task, the goal is to evaluate extraction quality, for which it is useful to look at precision /recal tradeoffs. Minor errors (e.g,, an additional word in non-categorical slot) should not significantly impact the readability of the extracted information. Second, in DST, queries are issued after most turns, so evaluating average performance at all turns makes sense. In contrast, in our IE task, information is accumulated (and corrected) for a final summary. In this case, turn averaging wveremphasizer earlier parts of a conversation. For that reason, our IE metric evaluates the full state (CB) at specific dialogue points (quarter, half, three-quarters, end), and turn averaging is used for evaluating the prediction of state changes (TLB).
Our task requires the scoring to handle multi-value and extended free-form text responses. For scoring purposes, we treat multi-value slots multiple instances of slot. For free- form values, we adapt the multi-span setup in Li et al. (2022) and enumerate all possible alignments between the predicted and gold values. Each gold value is aligned to one predicted value at most, and percentage match is computed based on the longest common substring (LCS) to give partial-credit score in [0, 1] (rather than requiring exact match, i.e, {0, 1} score) for use in measuring precision and recall.
Cumulative Belief (CB) State Scores (evaluating X) are computed for a particular turn (specific index & or dialogue-final turn) in the nth dialogue, denoted as mcs (1t,) 4²t m( nri, 's inn), where m can be precision (P) or recall (R). Overall scores Ninl R
are obtained by averaging over all dialogues Mi = {n: Rm + @ For example, precision is given by CB-P(f) M EncA; (n)
Turn-level Belief (TLB) Scores (evaluating X) are computed at the turn level, all of which are based on averaging over all N dialogues in the test set formulated as N E | LITT, MTYPr tn, where Tn {t Rm * Ø) and TYPE E {TLB, R, RS, sv) de- notes diagnostic score type. The scores (MITLB, Mg, MRs,, mşy) are described in Appendix For each turn, the MTLB is performance over the TLB; my how well referents are recog- nized; mss is how well referents are associated with slots when ignoring values; and msv is performance of slot-value detection when ignoring referents.
4 Datasets
We were provided with private dataset of 34 natural auto insurance claim calls (AIC). In each call, the agent's task is to gather detailed information about an auto accident. The calls were human transcribed and labeled using schema with six referents and sixty possible slots from ten domains (Appendix E.3). Calls had high variance in length and complexity, as shown in Table in Appendix B Additionally, 50% of dialogues had multiple values for at least one active slot. We split the calls into 7/4/23 for main/val./ter sets aiming for slot count split of 20/10/70.
Using AIC as target dataset for augmentation, we apply DIALGEN with ChatGPT (gpt-3.5-m) 5ott-3.--urbb--ee 5-tubbb----311 as the LM backbone to create DIALGEN-AIC, which contains 235 labeled dialogues (Appendix E.5 Reviewers completed one-hour training to become fa- miliar with the task and practiced generating one dialogue under supervision. Full training was complete after they received feedback for their first 3-5 dialogues. They were instructed to aim to generate dialogues with ze 50 turns. On average, each dialogue comprises 8 subdialogues, with 38% of turns receiving edits and 20% of turns being deleted. Each Saadiaaoousswwa
dialogue involves 9 + 10 times of partial or full subdialogue regeneration.
IIn the initial turns, there may be nothing extract and no false predictions; thus R. Ø.
6

---

Published as a conference paper at COLM 2024
DialGen AIC
AutoGen
AIC
AIC
dialogs
34
235
195
# turns / dialog
197
46
31
# tokens / dialog
4195
1128
947
user tokens turn
18
22
27
agent tokens / turn
25
27
35
Table 1: Statistics are calculated on the full dataset.
Data collection occurred over months with multiple iterations as documentation and task instructions evolved to become more comprehensive and consistent. The final version of the task instructions further encouraged workers to update slot values in multiple ways and include multiple values in slot (as described in $2.1). We follow the methodology in SQuAD (Rajpurkar et al. 2016), calculating anter-annotator agreement (IAA) at the turn level with three annotators and 32 dialogues, with resulting lAA of 78.5% Fj (Appendix E.2
We investigate the importance of including human reviewers in DIALGEN by comparing fully automatically created dialogues with prompting ChatGPT using the same training scenarios. We refer to this LM-only framework as AUTOGEN and it generate the AUTOGEN-AIC dialogue.. Prompt details are in Appendix D.2 In comparing AIC, DIALGEN-AIC, and AUTOGEN-AIC, we find that synthetic data has fewer turns, longer turns, and less variance in length, with fully automatic data being the most extreme. Adding human in the loop results in much longer and more varied dialogues, but they are still far from the complexity human-human dialogues (see Table 5 in Appendix B). Compared to MultiWOZ (Budzianowski et al. 2018), DIAL ATC is more complex. MultiWOZ dialogues average turns and 8 active slots per dialogue, compared to 46 turns and 38 slots on average for DIALGEN-AIC, and 198 turns and 48 slots for AIC. We split DIALGEN-AIC into train/val. tta./ sets with a ratio of 80/10/11 10 dialogues, selecting val./test sets by randomly sampling from the final iteration of data collection. Foolaccontain additional statistics of AIC, DIALGEN-AIC and AUTOGEN-AIC
These dialogues are generated by
5 Experiments
In-context Learning. Hu et al. (0022) propose IC-DST and use schema prompts and specialized retriever enable tew-shot in-context learning to predict state change with an LM. Given longer dialogues, a more complex ontology, and more slots to track than the datasets, the representation of dialogue history becomes a crucial concern. The SQL tables of the ontology is 1696 tokens, and our chosen LM (gpt-3. 5- t-ttuuroo-000 03010 has a token limit of 4096 tokens. To accommodate the token constraints, we truncate the in-context examples when given a longer dialogue state. We extract the TLB at the turn and accumulate the TLBs as CB. Details of the SQL table are shown in Appendix D.3
Furthermore, our task requires the model to identify the corresponding entity (referent) for the predicted slot-value pair. We redesign the prompt (Appendix D.3) to instruct the LM to generate the referent, slot, and value simultaneously. The retriever, SBERT (Reimers & Gurevych| 2019), is finetuned on the full DIAL training set, which is also used as the example selection pool. Due to privacy concerns, we only evaluate IC-DST on the DIALGEN-AIC test set.
Finetuned Transformers. We follow idea of the previous work (Lee et al.) 2021 Lu et al. 2024) to independently extracted the information and finetune T5 (Raffel et al. 2020) and Long-I5 [Guo al., 2022) with schema information embedded in the prompt. The models predict only active slots (together with referent and value) with one prompt per domain. The CB is the aggregate of predictions over all domains.
In addition, we explore four different configurations of prompt and model outputs:
Long-T5t: Use {(A,U) U= to predict CB.
7

---

Published as a conference paper at COLM 2024
Method
CB
CB1
CB₂
CB3
CB4
TLB
IC-DST
71.3
71.9
68.5
68.4
68.2
68.1
Long- 15t
71.8
72.5
71.7
71.0
70.4
Long-15 I5
66.3
64.3
64.8
64.3
63.9
68.5
T5
76.8
78.4
74.9
73.7
74.1
73.9
T5-SC
78.2
79.3
76.4
76.6
76.9
74.2
T5-SC$
78.5
78.7
76.2
76.0
76.2
75.0
Table 2: F scores on the DIALGEN-AIC test set. S denotes results with name substitution.
Long-T5: Use ((A,4))) to predict TLB; add to CB.
T5:
Use -) to predict TLB; add to CB.
T5-SC:
Use (-, and previous domain CB predict state change ACB; update CB.
Because the input length can be longer than 1k tokens, we choose Long-15 to cover all turns with the prompt, while the T5-based models make predictions based on the current turn only. T5-SC further considers the state change ACB, which is similar to the TLB but augmented with the four state-change commands. Details of the finetuning and in-context learning configuration are given in Appendix and details of the prompts for the different cases are given in Appendix D.4
Experimental Setup. When conducting experiments involving AIC, the model selection criterion is the highest TLB Fi score on the AIC validation set. For experiments solely on DIALGEN-AIC AUTOGEN-AIC, models were chosen based on TLB F1 score on the DIALGEN-AIC validation set. Additional hyperparameters can be found in Appendix C.1 All reported values represent the medians of five different random seeds.
Results
We report the main results on both cumulative and turn update scores. The cumulative scores are presented in two ways: CB as an average of CB across every user turn, and CBq as the CB at user turn f, where a dialogue. Thus, f will a specific turn, at either quarter, a half, three-quarters, or the end of the dialogue. The score of the last cumulative belief state CB₃ is the full F, score and can be regarded as evaluating a conversation summary. Model development was done only of real data.
[QT/4] Q E [1,2,3,4 4] and T is the total length of
the synthetic data to minimize Results on DIALGEN-AIC Test Set. The results on DIALGEN-AIC with different learning strategies and T5 configurations are presented in Table 2 The performance of IC-DST is lower than all T5 variants, although this may be due to the difference in use of domain- specific prompts. Note that our IC-DST implementation is based on the same ChatGPT model used for generating the DIALGEN-AIC, collaboration creates data sufficiently different from ChatGPT text such that ChatGPT cannot easily address this task. Predicting CB directly requires the full history, which is only possible with Long-T5. T5. With Long-T5, there is a benefit to predicting CB directly over TLB. However, optimizations needed to handle a longer history have tradeoffs that result in performance that is worse than the standard T5 model with TLB prediction for this task. T5-SC achieves the best result, which updates values rather than adding them new values in a list.
the low results suggest that human
To mitigate the potential risk of LMs generating personal information linked to randomly generated names in shared data, we replace them with other randomly generated names. As shown in Table 2, T5-SC exhibits comparable performance on both the original and renamed dialogues, indicating that the renaming process does not impact the model's effectiveness.
Results on AIC Test Set. The two best models (T5 and T5-SC) are used in experiments on the real data (AIC). The F results for different training sources are given in Table 3 The performance for the model trained on the synthetic data alone is better than with the small amount of the real data, but the best results are obtained by model trained on the combined
8

---

Published as a conference paper at COLM 2024
Method
Data
Caag
CB1
CB₂
CB3
CB₁
TLB
AIC
38.3
39.6
37.1
36.2
35.1
34.8
I5
DIALGEN-AIC
40.4
41.7
42.6
39.9
37.7
40.9
T5
AIC + DIALGEN-AIC
43.7
42.9
42.2
43.0
41.9
43.7
T5-SC
AIC
39.2
40.0
38.1
37.1
36.1
33.9
T5-SC
DIALGEN-AIC
41.0
43.6
42.1
41.3
40.5
38.9
T5-SC
AIC + DIALGEN-AIC
46.2
47.8
47.2
45.9
45.3
44.6
Table 3: Fj scores on the ATC test set for different training data.
AIC
DaGGe-AFE
AFC DaGGe-AIC
CB
20
T1B-41
Prodkca
Racil
Prekies
Clhys
Claas
ASC (1 1.33 turas)
Alc DiaGGes-AC
0.9%
1.3k
2.7k
3.5k
4,4k
5.3k
6.2k
7.1k
7.5k
8.8k
10%
30%
40%
50%
60%
70%
80%
90%
100%
and % the DialGen-Ae
Figure 4: TLB scores for T5-SC on AIC test set by varying the amount of DIALGEN-AIC training data.
Figure 3: CB precision and recall scores on the AIC test set and T5-SC models.
data. The difference between the best case model (T5-SC) on ATC alone in combination with DIALGEN-AIC is significant with < .01 using a bootstrap test. Because of the higher frequency of state changes the human-human dialogues, there greater benefit from the T5-SC model for the real data, with an 8% improvement in the CB₁ score compared to 4% for the synthetic data when using all training data.
To provide more insight into performance, we present the precision/recall results for CB in Figure Incorporating synthetic data yields higher recall and outperforms using real data alone in terms of Fi. The increased recall can be attributed to the inclusion of wider range of values in the synthetic data, which are not covered by the AIC training set. However, this improvement comes at the expense of lower precision. By combining both data sets, the model achieves better alignment with real-world data while retaining the advantage of high recall scores from the synthetic data.
We also experimented with varying the amount of synthetic data used in training the model in order to determine the relative value of synthetic versus real data. Figure 4shows that using 59 synthetic dialogues (approximately 2.7K turns) yields results similar to those obtained from the AIC training set, which consists of 1.3K turns in 7 dialogues. These results suggest that roughly 2.1 times as many turns of synthetic data is needed to match the performance of the real data, or 8.4 times many synthetic dialogues since the synthetic dialogues are shorter. However, the synthetic data is more valuable in combination with real data, for which the benefit beyond 97 dialogues (50%) is minimal. This suggests an opportunity for further improvement through strategic scenario sampling.
Comparison DIALGEN with AUTOGEN. To understand the importance of the role of humans in data synthesis, we investigate whether an LM annotator is sufficient and compare the value dialogues generated by AUTOGEN and DIALGEN. First, we use ChatGPT and GPT-4 (OpenAl. 2024) to annotate
training dialogues and compare the annotations with labels, obtaining 52.3 and 65.5 TLB accuracy respectively. This suggests GPT-4 is better annotator than ChatGPT, which aligns with the findings of

---

Published as a conference paper at COLM 2024
Framework
Dialogue Generator
Annotator
TLB
ChatGPT ChatGPT
ChatGPT GPT-4
49.5
AUTOGEN
60.7
ChatGPT w/ Humans
GPT-4
63.2
DIALGEN
ChatGPT w/ Humans
Humans
73.9
Table 4: Comparison between different data synthesis frameworks. The TLB F, is obtained by finetuning a 15 models on generated data and testing on the DIALGEN-AIC test set.
Gray et al. (2023) and Tekumalla & Banda (2023). Then, we finetune T5 models on each setting and report TLB score to evaluate the quality of synthesized data (Table 4). T5 trained on dialogues generated by the DIALGEN framework and annotated by humans yielded the best result, demonstrating how human-in-the-lopp data synthesis framework can provide higher quality training data than fully automated system.
6.1 Error analysis
Out of the 56 slots in the AIC test set, we noticed an improvement in 45 slots, while 4 slots were tied, and the remaining 7 slots have slightly worse performance. Our error analysis reveals two main categories for the performance loss: data mismatch between AIC and DIALGEN-AIC and over-reliance surface-level features.
Data Mismatch. We lose performance for the slot Car Mileage because of a difference in language used when describing the mileage of a car. In AIC, agents ask a binary confirmation for whether the mileage on the vehicle is above a certain threshold, whereas callers in AEAIII describe car mileage with an exact number. For the slot Traffic Controls Obeyed, AIC callers indirectly indicate that traffic controls are not obeyed, e.g. stating that the other driver ran a red light. DIALGEN-AIC AIC, the agent asks the caller to confirm directly whether traffic controls were obeyed.
Surface Level Text. The model both over- and under- sader--redicts slots due surface-level features such as predicting Number of Involved Cars when the text discusses counting ve- hicles, despite many such instances in AIC simply describing the traffic environment to contextualize the accident, e.g., there was vehicle in front of the caller, but it was not involved in the accident. The model also predicted this slot when there was language about the number of passengers with driver. Similarly, Color would be predicted whenever colors were mentioned, e.g., a purple bruise. Traffic Flow was severely under-predicted when it would have been beneficial for the model to predict the slot whenever saw information describing lane direction.
Conclusion
We propose DIALGEN, in which humans and LMs collaborate to generate long, complex dialogues. We demonstrate its effectiveness by synthesizing auto insurance calls and conducting information extraction (IE) experiments. While we build on the DST framework, our IE experiments target an ontology and data that are more complex than the DST task was originally designed for. To serve the IE task, we introduce an entity-centric scoring methodology more suitable for our IE task than the conventional joint goal accuracy metrics used in DST. In our controlled experiments, we contrast the outcomes of training with data synthesized through human-in-the-lon entirely by ChatGPT or GPT-4. Our findings indicate that human-LM collaboration enhances the process at both the data generation and annotation phases, validating the importance of including humans in the data synthesis process to generate more realistic dialogues. Our experiments demonstrate that the data generated by DIALGEN, despite dissimilarities with the data it is designed to emulate, can significantly improve model performance for information extraction on real-world human dialogues.
method against data generated and annotated
10

---

Published as a conference paper at COLM 2024
8 Limitations
While DIALGEN can be used to generate synthetic data for privacy-constrained settings, the effectiveness largely depends on the LM employed, target setting, and language. We con- ducted all experiments in the auto insurance claim calls domain in English, where English is high-resource language, and descriptions of car accidents are reasonably frequent in online text. An LM without reasonable capability in generating text in the target domain and language will result in low quality subdialogues, which can result in a frustrating collaboration for the human reviewers.
While DIALGEN results in dialogues that are more similar to real human-human interactions than AUTOGEN, our analysis shows that there are still substantial differences. This gap may be reduced by incorporating other forms human feedback, more powerful LMs, and or changing the prompt configuration. For example, subdialogue generation in DIALGEN is guided by including the full dialogue history as context for each subsequent subdialogue. LMs have finite context input length, so the max length of generated dialogue is limited by the chosen LMM Methods to overcome this limitation can include truncating the dialogue history context, investigating which parts of the prompt contribute little to guiding the LM, and representing dialogue history in a more efficient manner.
In our controlled experiments comparing AUTOGEN and DIALGEN, the dialogue generation and annotation process in AUTOGEN is accomplished by single LM agent with different associated prompts. There is room to improve AUTOGEN, such as exploring different prompt designs and introducing quality control agents.
Ethical considerations
Preserving privacy (Xin et al. 2020 Liu et al.. 2022b Torfi et al. 2022) is an important
challenge in synthetic data generation. Ensuring important characteristics in synthesized data with DIALGEN requires a domain expert who may have access to real, private data and can unintentionally leak information. DIALGEN-AIC, on the other hand, generates personal information using the Faker package@but there is potential for the LM to produce personal details related to randomly created hames. To mitigate the potential risk in shared data, we use gender guesser package to detect the gender of each name and replace it with other same-gender name. If DIALGEN users plan to publicly release their cata, they should remove potentially identifying information such as names from the synthesized data. In the released DIALGEN-AIC, we replace names with random alternatives to prevent the inadvertent generation of sensitive personal information by the LM.
Other than privacy issues, LMs can produce harmful content, and the risks of such pro- duction can increase depending on the target data setting. When employing humans to collaborate with LMs, practitioners should determine whether additional safety features such as toxic language filters are required to protect the workers.
Regarding the data collection hiring process, all dialogue reviewers were recruited from university listings and compensated at a rate of $18.69 per hour, following university practices. Prior to data collection, we instructed our reviewers familiarize them with the ontology, annotation guidelines, and criteria for assessing dialogue quality. We established Slack workspace for smooth communication with the workers throughout the process, providing feedback and promptly addressing questions and concerns they raised. This interaction ensured high quality of the gathered data.
Acknowledgments
We would like to express our sincere gratitude to Kevin Everson, Yanda Chen, and Yushi Hu for their invaluable discussions and preliminary studies. We would also like to thank Bing-
hatoss//ggihbb..m//okkk////ks http:p///tttiiiacccm//adaaaaraap
11

---

Published as a conference paper at COLM 2024
Syuan Wang and Irene Wang for their expert web programming consulting and debugging support. Additionally, we extend our appreciation to members of UWNLP for their valuable insights and contributions throughout the project. Lastly, we grateful to the diligent student reviewers from the University of Washington for their dedicated efforts in data creation. Their contributions were essential to the success of this research.
References
Tobias Albrecht, Theresa Maria Rausch, and Nicholas Daniel Derra. Call me maybe: Methods and practical implementation of artificial intelligence in call center arrivals' forecasting. Journal of Business Research, 2021. URL https://api. semanticscholast org/CorpusiD;
225140800
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth Interna- fional Conference on Learning Representations, 2024. URL http::tts////// //tp///evee//.eeee////ee net/forum?
id=hSyW5go@v8
Jianzhu Bao, Rui Wang, Yasheng Wang, Aixin Sun, Yitong Li, Fei Mi, and Ruifeng Xu. A synthetic data generation framework for grounded dialogues. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10866-10882, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: (0.18653/v1/2000 acl-long.600 608. URL
2wwwg/2223...m aww-llog...e.... 608
Helena Bonaldi, Sara Dellantonio, Serra Sinem Tekiroğlu, and Marco Guerini. Human- machine collaboration approaches to build a dialogue dataset for hate speech countering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, Pp. 9031-8049 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aalaanthoolgg org/
www.orgg//..eeeee 2022. emm-le-aimmp--m 549
Pawel Budzianowski, Tsing-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. MultiWOZ large-scale multi-domain Wizard- of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016-5026, Brussels, Belgium, October-November 2018, Association for Computational Linguistics. doi: 10.18653/vi D18-1547. URL /tLLitttss///aaaaooooogg.ooo 1otprtthooooo...r//1-115111/ 1547
Derek Chen, Howard Chen, Yi Yang, Alexander Lin, and Zhou Yu. Action-based conver- sations dataset: A corpus for building more in-depth task-oriented dialogue systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, pp. 3002-3017, 2021.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michae Frahm (eds.), Computer Vision - ECCV 2020, pp. 694-710, Cham, 2020. Springer International Publishing. ISBN 978-3-030-5852666.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7282-7296, Online, August 2021. Association for Computational Linguistics. doi: 10.18653 v1 v2/221 acl-long.565. URL htttps://aclaanhooloy..rrg wrgp://oooyoog//22//2/// acl- Long.long.500 565
Yao Dou, Maxwell Forbes, Rik Kediis-Ki Noah A. Smith, and Yejin Choi. Scarecrow:
A framework for scrutinizing machine text, 2021.
Yao Dou, Maxwell Forbes, Rik Honcell-edziorskee Noah A. Smith, and Yejin Choi. Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine
12

---

Published as a conference paper at COLM 2024
text. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. pp. 7250-7274, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: acl-long. 501
100.1663/v1/2022.aar /200222cl--oong.501 URL htttp:://aclanttologg.
Mor Geva, Yoav Goldberg, and Jonathan Berant. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPPIIJCN.P), pp. 1161-1166, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653 11/7 URL_htts::/aclaanthllogy
Morgan Gray, Jaromir Savelka, Wesley Oliver, and Kevin Ashley. Can gpt alleviate the burden of annotation? In Legal Knowledge and Information Systems, pp. 157-166. IOS Press, 2023.
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. Long15: Efficient text-to-text transformer for long sequences. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022, pp. 724-736, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653 /v1//v1/22222 findings-naacl.55.
10.18653/v1/2022 org htt.g///2........oooooooe findings-naall....
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Mar- ilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 2 (Short Papers), pp. 107-112, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653 11.18653vv//1186 11/1112211 URL https://accaattioooyyyor///
Sophie Henning, William Beluch, Alexander Fraser, and Annemarie Friedrich. survey of methods for addressing class imbalance in deep-learning based natural language processing- In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 523- 540, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/ /v1 http/////2222//// 12a.2222eaccmmmin.33 URL / httpss////ccaannooooo..oooro www.orggg/22....
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. In-context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2627-2643, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL htttps://aclantholo. org/2022. findings emnlp. 193
J. F. Kelley. An iterative design methodology for friendr natural language office information applications. ACM Trans. Inf. Syst., 2(1): 41, jan 1984. ISSN 1046-8188. doi: 141145/357417.557200 URL
01455/55/001111111177117111111.1 357420
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Álikhani, Gunhee Kim, Maarten Sap, et al. Soda: Million-scale dialogue distillation with social commonsense contextualization. arXiv preprint arXi:::212. 10465, 2022.
Sophia Lam, Charles Chen, Kristi Kim, George Wilson, J Holt Crews, and Matthew S Gerber. Optimizing customer-agent interactions with natural language processing and machine learning. In 2019 Systems and Information Engineering Design Symposium (SIEDS), pp. 1-6.
IEEE, 2019.
Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. Dialogue state tracking with language model using schema-driven prompting. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4937-4949, Online and Punta Cana, Dominican
13

---

Published as a conference paper at COLM 2024
Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v /v1 2021 meellemmlp-mann cacnelpp-mii.44440 main.404. URL/ittpss//accaattholgg. http:/...rrr//// .0021.eema .emolpp wwwnle-mainn 404
Haonan Li, Martin Tomko, Maria Vasardani, and Timothy Baldwin. MultiSpanQA:: A dataset for multi-span question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1250-1260, Seattle, United States, July 2022, Association for Computa- tional Linguistics. doi: 10.18653/v1 http////2222aaaaa// maacl-main.90. URL https:://aclanthoologg org/2022. naacl-maaimannn.. 90
Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large language models for text classification: Potential and limitations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10443-1046111 Singapore, December 2023, Association for org/ 2023 emnlp-main. 647
Computational Linguistics. URL httLittts///aaaanhholoe
Joseph CR Licklider. Man-computer symbissss. IRE transactions on human factors in electronics, (1):4-11, 1960.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6826-6847, Abu Dhabi, United Arab Emirates, December 2022a. Association for Computational Linguistics. URL https:// wwwpanthologg..og/2222222 foni22.fiinddi ema...gs-enn...555
Fan Liu, Zhiyong Cheng, Huilin Chen, Yinwei Wei, Liqiang Nie, and Mohan Kankanhalli. Privacy-preserving synthetic data generation for recommendation systems. In Proceedings of the 45th International ACM SIGIR Conference
Research and Development in Information Retrieval, SIGIR '22, pp. 1379-1389, New York, NY, USA, 2022b. Association for Computing Machinery. ISBN 4701450387333, 10. 11.114473477495 3532044.
doi: 10.1155/347495..553000 URL httpp:://oi... org/
Bo-Ru Lu, Yushi Hu, Hao Cheng, Noah A. Smith, and Mari Ostendorf. Unsupervised learning of hierarchical conversation structure. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5657-5670, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/ v1 htt/222..nninngeemmlllp l/aclantholoogy.rgggg org/0022 findings-emnlp 415
URL httss
Bo-Ru Lu, Nikita Haduong, Yhien-Yu Lin, Hao Cheng, Noah A. Smith, and Mari Ostendorf. Encode once and decode in parallel: Efficient transformer decoding, 2024.
OpenAL. Gpt-4 technical report, 2024.
Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interfuce Software and Technology, pp. 1-18, 2022.
Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim. Data synthesis based generative adversarial networks. Proc. VLDB Endow. 11(10):107111033 jun 2018. ISSN 2150-8097. doi: 10.14778/32317751.1
URL
1ati///2..1177//2311555 3231757
Piotr Pęzik, Gosia Krawentek, Sylwia Karasińska, Pawel Wilk, Paulina Rybińskk, Anna Cichosz, Angelika Peljak-Lapinska, Mikotaj Deckert, and Michal Adamczyk. DiaBiz an annotated corpus of Polish call center dialogs. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference, PP. 723- 726, Marseille, France, June 2022. European Language Resources Association. URL 1rec-1.76
http:://aclanthhoogg..r/////////
14

---

Published as a conference paper at COLM 2024
Kun Qian, Ahmad Beirami, Zhouhan Lin, Ankita De, Alborz Geramifard, Zhou Yu, and Chinnadhurai Sankar. Annotation inconsistency and entity bias in MultiWOZ. In Haizhou Li, Gina-Anne Levow, Zhou Yu, Chitralekha Gupta, Berrak Sisman, Siqi Cai, David Vandyke, Nina Dethlefs, Yan Wu, and Junyi Jessy Li (eds.), Proceedings of the 22nd Anmual Meeting of the Special Interest Group Discourse and Dialogue, pp. 326-337, Singapore and Online, July 2021. Association for Computational Linguistics. doi: 10.18653 11186653/1/222111 sigdial-11355 URL httph/tss//aaaaathllloo/o//o/ org 202gg2221..22 sigdial-1.35
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 2011863/vv/D16-111 ittps://aalanntooley..rr//111111
URL
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, number 05, pp. 8689-8696, 2020.
Nils Reimers and Iryna Gurevych. Sentence-BERR Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMÑLP-IJCNLP) pp. 3982- 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.1860.1///// (00.18633v11/D1/ 1111/111/111110...
Ashish Sharma, Inna Wanyin Lin, Adam S. Miner, David C. Atkins, and Tim Althoff.
Human-ai collaboration enables more empathic conversations in text-based peer-to- peer mental health support. Nature Machine Intelligence, 5:46-57, 2022. URL https
7/api. peapiissmmaticcchoo..o htt.:/aaann...ccoooooooooooooror
Felix Stahlberg and Shankar Kumar. Synthetic data generation for grammatical error
correction with tagged corruption models. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 37-47, Online, April 2021. Association for Computational Linguistics. URL
1.
Ramya Tekumalla and Juan M. Banda. Leveraging large language models and weak supervision for social media data annotation: An evaluation using covid-19 self-reported vaccination tweets. In Hirohiko Mori, Yumi Asahi, Adela Coman, Simona Vasilache, and Matthias Rauterberg (eds.), HCT International 2023 Late Breaking Papers, pp. 356-366, Cham, 2023. Springer Nature Switzerland. ISBN 178-3-031-48044-
Amirsina Torfi, Edward A. Fox, and Chandan K. Reddy. Differentially private synthetic medical data generation using convolutional gans. Information Sciences, 586:485-500 2022. ISSN 0020-0255. doi: https:/ 10tp:/001..101/......1...2.2.0.. //www.cceencddiress httpe//cwwccccceeeeee////e//e//a
URL https
Adaku Uchendu, Jooyoung Lee, Hua Shen, Thai Le, Dongwon Lee, et al. Does human collaboration enhance the accuracy of identifying llm-generated deepfake texts? In Proceedings of the AAAI Conference on Human Computation and Crowdsourccnng volume 11, pp. 163-174, 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484-13508, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 /222222cc.looonge.... URL https:: httpss:/aalaatholggg.oo org.o/y....gg/23/
acl- 754
15

---

Published as a conference paper at COLM 2024
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing- In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. URL https: www. aclweb. org/ anthology/2024 2020. emnlp pelp--eemo..e 6
Bangzhou Xin, Wei Yang, Yangyang Geng, Sheng Chen, Shaowei Wang, and Liusheng Huang. Private fl-gan: Differential privacy synthetic data generation based on federated learning. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), PP. 2927-2929999222 2931, 2020. doi: 10.1109
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Zhipeng Zhao, Kun Zhou, Xiaolei Wang, Wayne Xin Zhao, Fan Pan, Zhao Cao, and Ji-Rong Wen. Alleviating the long-tail problem in conversational recommender systems. Tn Proceedings the 17th ACM Conference on Recommender Systems, RecSys '23, pp. 374-385, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702419, doi: 10.1145/360995..666
URL https: https:://oi..rg/// org.oi.orgg/e 11453/66911119/ 3608812
16

---

Published as a conference paper at COLM 2024
A Evaluation metrics
As described in subsection 3.3 we denote the equations of turn update scores for four diagnostic score types TYPE E (LBB R RS,sv). All of which are based on averaging over all N dialogues in the test set formulated
1
Σ TET
(n,
N
Tm
where Tn - (t: Rxt + Ø).
Specifically, the equations of the specific scores (πTTYEε) are follows,
MILB (1, = Rnt rERa.
Σ
m(Snnt, Sin))
mx mR(n,t) m (Rnt, Rūt)
(n, 1
Σ
m(Smit
|Rn
Ra
Msy (sy( m
U
Sunt, U
S
rER
rERM
where Sart the set of slot labels associated with referent in of the n-th dialogue.
B
Data statistics
AIC
DIALGEN-AIC
AUTOGEN-AIC
dialogue turns/ / dialogue tokens / dialogue
34
235
195
197 + 98
46 8
311 7
4195 1: 2404
1128
230
947=214
user tokens/ / turn
18
17
27=17
agent tokens/ / turn referent-slot pair unique referent-slot referent-slot pair / dialogue /
25 31
27 = 14
35=16
1622
8844
109
152
48 24
38 8
% dialogue w/ updates % dialogue w/ multiple values
50.0%
14.5%
50.0%
19.1%
Table 5: Statistics are calculated on the full dataset. Tokens are calculated with Huggingface T5 tokenizer.
Table showw the detailed statistics of three datasets: AIC, DEAIIGN- AUTOGEN-AIC For AUTOGEN, we only generate and annotate the training set with LMs; thus the statistics based on human annotations are not available.
C Training and generation details
C.1 Finetuning details
All experiments are done with T5-base or Long-T5-bass with Huggingface implementation (Wolf et al., 2020). Training time for full DIALGEN-AIC and AIC setting is averaged 3 hours on 2 NVIDIA V100 GPUs. For the experiments on only DIALGEN-AIC, we use NVIDIA A40 GPUs. The total number of GPU training hours is 110 hours.
17

---

Published as a conference paper at COLM 2024
Hyperparameter
SBERT
or T5-SC
Long-T5
Training batch size
Learning rate
2 x 10-5
5 x 10 10-4
x 10-4
Max generation length
256
256
Max input length
512 Pretrained Checkpoints all-mpnet-bers base-v2
512
2592
t5-base
long- taw-t---tglll-lbbee
Table 6: Hyperparameters for training in-context learning retriever SBERT and finetuned models T5, T5-SC and Long-T5. The other parameters are default values in Huggingface trainer.
C.2 Hyperparameters of OpenAl API calls
Hyperparameter
DIALGEN
AUTOGEN
LM annotators
IC-DST
Language Model
ChatGPT 0.85-0.9
ChatGPT 0.0
ChatGPT or GPT-4
ChatGPT
Temperature
0.9
0.0
Max tokens
512
256
2048
512
Stop strings
[I<\div>>]
["-", "An", n. *#]
Presence penalty
0.2
0.0
0.2
Frequency penalty
0.2
0.0
0.2
Table 7: Hyperparameters for API calls. We use 3pt-3.5-turbo- and gpt-4-0613 for ChatGPT and GPT-4 respectively. For AUTOGEN, the stop token is the default value of the API call. We parse the output of the response from ChatGPT and truncate tokens after the string [End of conversation].
D Prompts
We show the prompts used in DIALGEN and AUTOGEN for generating DIALIEN-AIC, AUTOGEN-AIC,IC-DST) T5, T5-SC and Long-T5 in the following subsections.
Instruction
Count
Have CALLER ER describe more car accident details with complex wassniing that involves two cars' motion.
23
Have CALLER's response be less specific. have AGENT asks for more details.
18
Splat AGENT's questions
multiple
18
CALLER's respetse less specific. have AGENT asks more details. AGENT a question accident
Have AGENT ask for permission to record the call.
15
Ask for email address and home address
14
Have CAI I ER ask AGENT questions about her insurance coverages multiple turns
13
Have AGENT ask CALLER more questions about the accident details
12
misremember the details. AGENT double check with CALLER. Explain coverages
12
12
Have CALLER corrects wrong information. have AGENT asks for clarification.
12
Break this convenstion down into multiple turns of dialogue
11
information Have AGENT for multiple turns
10
Break these turs
forth dialogue
10
AGENT needs split up her questions.
10
Table 8: Instructions with a frequency of 10 or more times used by humans to regenerate subdialogue.
Personality
Description
Aggressave
Feeling angry and confrontational about the accident, may place blame on others or use aggressive language. Focused the details and logistics the claim process, may ask for precise information and explarationns. Ursure about what happened during the accident what de
Analytical
Confused
ask lot questions.
Cooperative
Willing work with insurance company and other parties invalved resolving the claim.
Defensive
Feeling the
actions place blame Expertencing strong emotions related to accident, may crying or straggling ling maintain composure during call. Hesitant provide information or answer questions about the ae may be trying to conceal something.
justify
may unwilling
responsinility for the accident.
Emotional
Evasive
Impalient
Feeling frustrated with the claim process or the speed which is progressing, may express inritatio or urgency in their language. Trying to maintain positive and optimistic autlok during the call, may express grätitude for the assistance heing provided. Feeling distressed or frustrated due the accident and its consequences.
Ressouring
Upset
Table 9: The list of the predefined callers' personalities.
18

---

Published as a conference paper at COLM 2024
D.1 DIALGEN prompts
Prompts for instructing a language model to generate a desired subdialogue. Tabll 8) shows the instructions used by humans when regenerating a subdialogue. The top four instructions instruct the LM to produce more content, which naturally results in lengthier dialogues.
Prompts for providing callers' personality. TTall shows the list of predefined callers' personality. We randomly select one personality when creating the dialogue scenario.
Prompts for providing scenarios and dialogue content to generate a subdialogue. We present an example prompt used to generate the first subdialogue when using DIALGEN- AIC for auto insurance claim calls. It includes a task description, entity-slot-value triplets, an accident story, caller's 's and agent's personalities, and an initial exchange. Similar to Park let al.] (2022), we use HTML tags to denote different dialogue elements, i.e. <p> for turns and <div> for the subdialogue.
<short_ <short_summar>
story
Bob Parkhurst had busy day at work, and all he wanted to do was to go grocery shopping. As he backed out of her Office Depot parking lot, failed to notice the gray MAZDA B-Series Extended Cab driven by Spencer Tullar he turned into the same aisle from the opposite direction.
parking spot
Spencer, who was on his way to run sone errands, had been driving down the parking lot in extremely slow speed backing out of his spot. didn't think nuch of it and was about to just
when suidenly he Bob's 's yellow drive behind her when, at the last minute, he noticed that Bab seemed to be backing out without looking around. Spencer slamed passenger side of Bob's car.
his brakes, but it was late. The front right of his truck smashed hard into the back
The impact of the collision coused 's car spin around cone
stop. inmediately felt sharp pain in her neck and knew that sonething was wrong. As he tried to get out of the car, he realized that he couldn't nove his neck without experiencing excruciating pain. Spencer got of his truck and approached Bob' 's car, he asked if Bob okay. Bob told him that he was hurt and needed medical attention. Spencer called 911 innediately while also trying his best to confort Bob until help arrived.
When energency services arrived shortly after, they Found Bob slunged over in her sest, clutching his neck in brace around hin so wouldn' move head
agony. The respenders helped her out the car and placed while they examined her injuries. They then transparted him by anbulance to the hospital for further nedical attention.
Heanwhile, police were already their way.
arrival at the scene, they took statements from both drivers well as any witnesses who nay have seen what happened. Unfortunately, no one at the time had a clear view of the incident,
drivers agreed
they didn't t see each other before collision.
Since both cars were still in the parking lot when the accident hagpened, there was no need to redirect traffic. incident site to prevent any further accidents.
However, officers still to direct people away They also checked Spencer's license and found that it was valid.
The investigation into what caused the accident was inconclusive. Neither driver was certain about who was at appeared be
fault, they both believed the other driver failed observe their movenents. Since fault, no tickets or
entity-slot-value triplets
Accident details: (accident location, affice depot parking let), (danage part, unsure), num passengers, witnesses, date of accident, tine of accident, subjective fault, airbag deployed. Evidences of the car accident: police report, (pictures, no picture), police report number, police department name, tickets citations.
Traffic condition: weather visibility, (obstructions to view, no).
Caller's 's driver action: car notion, speed, traffic contruls obeyed, turn signal, (horn, no). Caller's car information: (nake/nodel, dadge stratus), nake year, color, car nilnage.
Caller's s injury details: body part injured, injury type, nedical treatment.
task description
Have role play car accident clain call. One person is agent Alice from car insurance comрany the other is the caller Beb who wants to file clain. beginning
call, have Alice ask Bab's permission record
proceeds
conversation.
Mithin sone <p> </p>, have simulate poor phone connection. Have Alice and Bob can not hear each other and need to repeat what they said.
Have Alice verify Bob personal information to access account information at the beginning of the call. car accident by using story tuples above to describe the accident. Have Alice confirm new information with Bob during the call to ensure consistency. each ather.
Bob describe
Have Alice Bob engage in small
Have Alice explain the insurance coverages to Bob.
personality
Bob is inpatient, feeling frustrated with the clain pracess or the speed at which it is progressing, ma express
irritation or urgency their Language.
Alice is conversational, personable, patient, empathetic, symptthetic and professional.
19

---

Published as a conference paper at COLM 2024
instructions
Use the story, information, and iersonality to create role play script and fallow the task description.
</short_sumaary
<div>
cp class="Alice" titie="Auto Accident"> Thank you for calling! This is Alice. How may I help you today? c/pp class="Bob" title="Auto Accident"> Hella. This is Alice. calling for accident. </p> </div>
Alice question car accident details.
<div>
D.2 AUTOGEN prompts
We use two separate prompts for generating an AUTOGEN-ATC dialogue and doing auto- matic annotation.
Prompts for automatic dialogue generation. The AUTOGEN-AIC prompt requires two things: scenario and a demonstration dialogue. The demonstration dialogue provides the language model (LM) the format and style of the desired dialogue, and the scenario describes the essential information the generated dialogue should cover. We select a dialogue from the DIAIGEN-AIC training set as the demonstration dialogue and share it across all generated dialogues. To compare DIALGEN and AUTOGEN, we use 195 scenarios from DIALGEN training dialogues and apply them to AUTOGEN to automatically generate dialogues. The LM generates a full dialogue following the format the demonstration dialogue and the information in the scenario. We cut the generated output after the phrase [end of the dialogue]. The output before the cut the final generated dialogue. The following python string is used the template.
Scenario'n (o((mmarai
Deno Hialoguenn(eenooceelle.
Lend of the dialogue]
Generate dialogue between the scenario.
for accident clain based the information mention
Follow the format style the demo dialogue.
Agent:
Prompts for automatic annotation. We use ChatGPT or GPT-4 automatically annotate pair of system user turns. [SYSTEM TURN] and [USER TURN] are the placeholders for user and system turns.
Possible Entities: Global, caller, Other Driver, Caller's Passenger, Other Driver's Passenger, Mitness. Entity description:
Global: it is for slot values that are not assaciated with entity.
Caller:
values
associated with caller.
other Driver: the slot values that associated with the other driver.
Caller's ss Passenger: the slat values that are associated with the caller's passenger.
other Driver's 's Passengar: slot values that
the other driver's passenger. Witness: the slot values that are associated with the witness.
Schena table:
donsin-slot possible values
Carinfo-Hake//adde non-categorical values CarInfo-Wake Year non-categorical values Car Info-Color non-categorical values
CarInfo-Car Hileage non-categorical Barlnoo-Ridshare (Uber/Lyft) yes, unsure AdjusterEExlaan Coverages non-categorical values Mejuster-Periissionn to Record
no
Adjuster-Set up Inspection photo clain, field assigmment
Adjuster-Set up Rental yes, no
TrafficEnviromentt wtw.Ewiviroeeeteettttee Visibility clear, cloudy, rainy, snowy, foggy, windy, other, unsure
rrafficinirrmmeettBttrrr
View yes, na, unsure
prafiicenviromeent-ne Condition dry, wet, slippery, debris, potholes, straight, curved, tunnel, steep incline, flat, other, unsure
Prafiicfiviroment-Tue Signal no signal or sign
sign, yield sign,
light, yellow light, red light, other,
TrafficEnviromments Dar.rooneenn-eeerppttoon
nornal, turn lane, shoulder. other, unsure
PerFiiEEviiooneet-aue of Lanes 1, 2, 3, 4+, unsure
TrafficEnvirommett Condition heavy, moderate, light, other, unsure
20

---

Published as a conference paper at COLM 2024
TrafficEnviromenn--e Linit non-categorical
Saafiicnviromme---rfffe. Flow one-way, two-way, other, unsure
trafficanvtrommet-Parkiii Type angled, straight, other, unsure
aclidettDetalls-Dmmeee Fart front, right, back, left, front right, front left, back left, back right, other, unsure
accidmt-ettill--eccidm Location parking lot, driveway, highway, roadway, intersection, other
AccidintDotails--um of Passengors 8, 1, 2t, unsure
MacidemtDttsill--iinnsssss yes, no.
Accidmnthetails--im of Involved Cars 1, 3, 4t, unsure
AccidenttDttaill=-hhhleee Involved yes, no, unsure
AccidentDeteill-AArrbe Deployed
unsure
AccidmmtDettsils-TToee yes, no, unsure
AcciddmtDDtaails-Podastrit Involved yes, no,
AccidantUettill--0ate of Accident non-categorical values
AccidentDetaills-iine Fault caller, other driver
non-categorical values
secidantintalll-subjeeeiive
Centactinfo-First Name non-categorical values
ContactInfo-Last Nane non-categorical values
ContactInfo-Hlome Address
non-categarical values
InntactInfo-Phons Number
ron-catagorical values
Contactinfo-Email Address
non-categorical values
Contactinfo Policy
non-categorical values
ContactInfo-Date of Birth
non-categorical values
DriverActions-Car Motion
traveling forward, backing, turning, changing lanes, stopped, other, unsure
triverActions-Sper non-categorical values
suerActions-Distraee
cellphone, animls, snoking, passengers, traffic, eating, not paying attention,
unsure, distraction
DriverActions- yes, no, unsure DriverActions-leen yes, ne, unsure
Dri privercctions-Tur Signal yes, unsure Dri Levverctiions-rrffii Traffic Controls Obeyed yes, no, unsure Destininnntion of Trip non-categarical values
Trip-Purpase of Trip non-categorical values
Trip-Origin of Trip non-categorical values
injuryDetails-Arbulee yes. no, unsure
injuryOetails--i InjuryDetalls-Injury injury
Injured head, neck, shoulder,
back, 1inb,
bruise, broken fracture, scratch, bleeding, strain sprain, other,
jjjrrbetailss-Mdicca Treatnent surgery, CAT hospitalization, ER, x-ray, other
Evi dences-Police Report yes, no, unsure Department weideees-Ticketes at scene, after accident, no picture, unsure party cited, party cited, no party cited, miltiple parties cited,
Evi
non-categorical
Ev
unsure. ticket
Evi Wridenee-PPllce Report Number ran-categorical values
Evi dences-Skid Yarks yes, no,
Example 1
system turn: Great, you providing information, Can you please me about accident? When and where did it happen?
user turn: The accident occurred at 5th Avenue and Main Street yesterday around 4 pn. I was driving on Main E350 Super Duty backed from lane and hit my car's right side.
Street
tlb label:
Locatian": E
II Intersection'
"AccidentDetai Is-Date of Accident": "Global yesterday"
"Accidenteetails- Accident":
"Global II pn."
DecileentDetall--Deaggg Part": [
"Caller II Right"
"CarInfo MorKneo-Makk/Model"": [
"Other
Super Duty Carge"
1.
"DriverActions-Car Motion":
"Caller Traveling Forward",
"Other Driver II Backing"
Descripptt of Lanes":
}
21

---

Published as a conference paper at COLM 2024
systen turn: Absolutely. Patrick. accident. You also have censsured/unerrnnsurre notorist coverage in case the other driver doesn't 't have enough insurance to cover accident. And finally, theft or natural disasters.
collision insurance, which cover danages your car
danages. Your MedPay coverage will pay for any nedical expenses resulting from the
have comprehensive insurance, which covers danages not related to collision, such
That sounds gand, Debra. Can you tell about ModPay?
tlb label:
"Adjuster-Explain coverages"; [
"Global II You have collision insurance, will cover danages your car from accident. You also have
notorist coverage case the other driver doesn 't have enough insurance cover the danages. Your MedPay coverage will pay for any nedical expenses resulting from the accident. And finally, you have comprehensive insurance, which danages not related collision, such theft or natural disasters."
Follow the demonstration label the slat values for the following systen and Only return extracted slot values. Use double quotes for keys and values in the returned dictionary. Return all labels in single dictionary.
dictionary.
system turn: [SYSTEM TURN]
turn: [USER TURN]
tlb label:
D.3 IC-DST prompt and output
Due to the input length limit, we extract the TLB at turn and accumulate TLBs as CB. Thus, [context] is empty.
CREATE
TABLE AccidentDetails(
Damage Part' TEXT CHECK ('Danage Part' IN "Frant', 'Right', "Back', 'Left', "Front Right', 'Frant Left', Back Left', 'Back Right', 'other', 'Unsure'), 'Accident Location' TEXT Intersection', 'Other'),
('Accident Location' 'Parking Lot', 'Driveway', 'Highway', 'Roadway',
('Nun Passengers' Ni Witnesses" TEXT CHECK ('Mi tnessess IN Yes',, 'No', 'Unsure"),
Passengers'
'1', '2+', 'Unsure),,
Nuum of Involved Cars' TEXT CHECK ('Nun of Involved Cars' IN '1', '2', '3", '4+', 'Unsure') ". 'chi Involved' TEXT CHECK ('Children Involved' IN "Yes" 'No', 'Unsure'), 'Airbag Deployed' TEXT CHECK (' ('Airbag Deployed' IN 'Yes", 'No", 'Unsure"),
'Towed' TEXT CHECK ('Towed" "Yes", 'Ne", 'Unsure').
Pedestrians Involved' TEXT CNECK ("Pedestrians Invalved' IN "Yes", 'Ne", 'Unsure'), Accident' TEXT, 'Tine of Accident' TEXT,
SSbjecttvve Fault' TEXT CHECK ('Subjective Fault! IN "Caller', 'Other Driver'),
CREATE Adjuster( Explain Coverages' TEXT,
CHEC ("Pernission Record' "Yes', 'No'), 'et Inspection' TEXT CHECK ('Set Inspection' IN "Quick Photo Clain', 'Field Assignment'), 'Set up Rental' TEXT CHECK ('Set up Rental' IN "Yes", 'No") N,
PPrri
CREATE CarInfo( 'Make/Model' TEXT, Make Year' TEXT,
"Color' TEXT,
"Car Mileage' TEXT, )
'Rideshare (Uber/Lyft)' CHECK ('Rideshare (Uber/Lyft)' 'Yes", 'No', 'Unsure'),
CREATE TABLE Contactinfo( 'First Nane' TEXT,
'Last Namre' TEXT.
Home Address' TEXT,
Number' TEXT,
Enail Address' TEXT,
Pallicy Number' TEXT,
Date of Birth' TEXT,
)
CREATE TABLE DriverActions(
('Car Notion' 'Traveling Forward', 'Backing'. 'Turning' 'Changing Lanes', Stapped' 'Other', 'Unsure'),
Motion'
'Speed" TEXT,
22

---

Published as a conference paper at COLM 2024
DDsttractions' CHECK ('Distractions" "Celiphene" 'Aninals', "Snoking", 'Passengers', 'Traffic', Eating', 'Not Paying Attention', Other", 'Unsure', ', 'No Distraction'), 'Brake' TEXT GHECK ('Brake' "Yes" 'No'. 'Unsure'),
'Horn' TEXT CHECK ('Horn' IN "Yes", 'No" 'Unsure'),
'Turn Signal' TEXT CHECK ("Turn Signal' IN "Yes". "No", 'Unsure').
'Traffic Contrals Obeyed' TEXT CHECK ('Traffic Controls Obeyed IN 'Yes', 'No", 'Unsure'
CREATE TABLE Evidences(
PPolice Report' TEXT CHECK ('Police Report' "Yes", 'No", 'Unsure'), PPllic Department Name' TEXT, 'Pictures' TEXT CHECK ('Pictures' IN "At Scene', "After Accident', "No Picture', 'Unsure')
'Tickets Citations' TEXT CHECK ('Tickets Citations' IN "Caller Party Cited', 'Dther Party Cited', 'No Party Cited', 'Multiple Parties Cited', 'Unsure', 'No Ticket'),
Report Number' TEXT, 'Skdd Marks' TEXT CHECK ("Skid Marks' IN 'Yes', 'No". Unsure'),
)
CREATE TABLE InjuryDetails(
Anbulance' TEXT CHECK ('Anbulance' "Yes", 'No', 'Unsure')
Body Part Injured' TEXT CHECK ( ('Body Part Injured' IN 'Head", 'Neck', 'Shoulder', 'Chest', 'Abdomen', 'Back', 'Linb', 'Other'),
'Injury Type" TEXT CHECK ('Injury Type' IN "Bruise', "Broken Fracture', "Ct Scratch', 'Bleeding', 'Strain Sprein', 'Sore', 'Other', 'No Injury'), 'Mdiica Treatment" TEXT CHECK ('Hedical Treatnent" IN 'RR', "Surgery', "at Scan', 'Hospi tolization', 'ER', X-Ray', 'Other'),
CREATE TrafFicEnwiromnett
Weather Visibility' TEXT CHECK ("Weather Visibility' "Clear', 'Cloudy', 'Rainy', 'Snowy', 'Foggy', 'Windy
"Other", 'Unsure"),
dostructions View TEXT CHECK ("Obstructions to View IN "Yes', 'No', 'Unsure'), Road Condition' TEXT CHECK ('Road Condittion' "Dry", 'Met', 'Slippery', 'Debris', 'Potholes'; 'Straight', Incline', 'Flat', 'Other', 'Unsure'),
"Durved' 'Tunnel',
"Traffic Signal' TEXT CHECK ('Traffic Signal' IN "Stop Sign". 'Yield Sign', 'Green Light', 'Yellow Light", Signal Sign"),
Light', 'Other" 'Unsure'
Descriptiin of Lanes" TEXT CHECK ('Description of Lanes' IN "Normal', 'Turn Lane', 'Shoulder', "Other", Unsure"),
Num of Lones' TEXT CHEOK ('Nun of Lanes' IN '1', '2', '3', '4+', 'Unsure'),
'Tafffic Condition' TEXT CHECK ('Traffic Condition' IN "Heavy", 'Moderate', 'Light', 'other', 'Unsure'),
"Speed Linit'
"Traffic Flow' TEXT CHECK ('Traffie Flow' IN "Wne-Way", 'Two-May', 'Other', 'Unsure").
"Parking Type" TEXT ( ('Parking Type' IN 'Angled', 'Straight', 'Other', 'Unsure'),
GREATE TABLE Trip(
Destination of Trip' TEXT,
PPrrpose TEXT,
OOrigin of Trip' TEXT,
Ussing valid sQuite, answer the follawing multi-turn conversational questions for the tables provided above.
Example
[context]
[systen] I see. Thank you for letting ne know. Can also provide ne with the make, nodel, and year of your well color?
[user] Of course. It's white Lexus sedan, 2018 model.
sQL: SELECT FROM CarInfo WHERE Caller-Make_Ven 2018 AND Caller-Color white AND Caller-Hake/Modell Lexus sedan,;
Example #2 [context]
[systen] Thank you for sharing that information, Lynne. Can also provide with the nake and model of your car?
model Toyota Camry. It's nodel, and had about 40,000 niles on it at the tine of the accident
[user] Yes,
white sedan.
SQL: SELECT FROM tarinfo WHERE Caller-Color = white sedan. AND Cal lor-Hake/MMddl Toyota Canry. AND Caller- Year_Year 2818 AND Caller-Car MMreeeeggeeeee 40, 968;
Example 43
[context]
[systen] see. Can describe your car's make and nodel? what year was it nade? And what color was it? [user] It's a white sedan, a 2018 Honda Accord. SQL: SELECT FROM CarInfo MIERE Make--Makk/Mddee sedan, 2818 Honda Accord. AND Caller Year 2018 AND Caller-Color white;
23

---

Published as a conference paper at COLM 2024
Example 84
[context]
[systen] Do remenber the nake and rodel of the other car?
[user] think it was black sedan, but I'm not completely sure. SQL: SELECT FROM CarInfo MHERE Dire-_Driver--kkk/WMddt sedan, AND Seher_Driver-Color- black;
Example #5 [context]
[systen] Thank you for that infornation, Joel. Can please provide with car's nake and model, year, color, and approxinate mileage? [user] Sure,
white sedan. It's 2016 nodel with aperoximately 50,000 miles it. SQL: SELECT FROM tarinfo MHERE caller-Hake/Madel sedan, AND Saller-Car_Mileago approxinately 5e,000 miles Caller-Hoke_Y:
Caller-Color
2016 model;
Example
[context]
[systen] Thank you for all the dstails, Richard. you please provide with your car's nake and model?
Q: [user] Yes, it's white sedan, 2007 make.
SQL: SELECT
CarInfo WHERE Caller-Color white sedan AND Caller-Make_ = 2087 FROM CarInfo MHERE Caller-Color white sedan AND Cakler-Meke Year 2807
FROM CarInfo MIERE Caller-Color white sedan AND Yeleer-Huke_Year 2887
D.4 Prompt and output for finetuned models
The previous study (Lee et al. 2021 employs independent decoding with natural language prompts for optimal outcomes. However, this approach necessitates the enumeration of all potential combinations of domain-slot pairs during both training and inference. As the ontology grows larger, the computational burden increases linearly. To address this issue, we propose to group slots with the same domain and train the models to predict all active slots with their values and referents simultaneously.
Long-T5 for CB prediction. We present a training example for the "ContactInfo" domain with complete dialogue history at time f. The example contains separators [s], [rv], and [srv] that label prior information as slot, referent-value pair, or slot-referent-value triplet, respectively.
Input:
policy number 123456789. [SYSTEM] Thank you, Could you please provide with your name and policy number sa I can access your account information? [USER] Yes, that's 's fine. [SYSTEM] I an so sarry that happened. Before begin, may please have your permission record training purposes? [USER] Hello. This Bob. I calling for a car accident. [SYSTEN] Thank you for calling AllState! This Alice. How may help you today? [donain] ContactInfo [possible slots] First Name (the First Name of the ContactInfo) [s] Last Name (the Last Name the ContactInfo) [s] Hone Address (the Home Address of the ContartInfo) [s] Phone Number (the Phone Number of the ContactInfo) [s] Enail Address (the Email Andress of the Contactinfo) [s] Policy Number (the Policy Number the Contactinfo) [s] Bate of Birth (the Date Birth the ContactInfo)
[USER]
Bob Lee,
call
quality and
Output: First Name [srv] Bob [rv] Caller [s] Last Nare [srv] Lee [rv] Caller [s] Policy Number [srv] 123456789. [rv] Caller
Long-T5 and T5 models for TLB prediction. We present a training example for the "ContactInfo" domain with the most recent two turns (A, U): time f.
Input:
[USER] H1, my nane is Bob Lee. I was recently in a car accident and wanted to file clain. [SYSTEM] Thank you you today? [donsin] ContactInfo [possible slots] First Name of the ContactInfo) [s] Last Nane (the Last Nane of the ContactInfo) [s] Home Address (the Hone Address of Contactinfo) [s] Phone Mumber (the Phone the Contactinfo) [6] Policy Number (the Policy Nunber of the ContactInfo) [s] Date of 8irth (the Date of Birth of the Contactinfo)
for calling!
may
(the First
the Contactinfo) [s] Email Address (the Email Address
Output:
First [srv] [rv] Caller [s] Nate [srv]
[rv] Caller
In the example, the caller (USER) mentions the first and the last name that are under the domain ContactInfo. The model is require to generate the active slots "First Name" and "Last Name" with the corresponding values "Bob" and "Lee", and referent "Caller."
24

---

Published as a conference paper at COLM 2024
T5 with State Change (T5-SC). For T3-SC, the models need to predict entity-slot-value triplets and edit operations associated with the triplets. The final output of a state at time t will be calculated by applying the edit operations on the associated triplets given the previous state at time 1 - 1. We consider four edit operations: [new], [keep], [delete], and [concat]. We describe the four edit operations in the following paragraph.
If triplet has not been observed in the previous state, the model is expected to predict [new]. Conversely, if the triplet has already been mentioned in the previous state, the model must predict [keep]. The [delete] operation is employed when a triplet mentioned in the previous state should be removed. If the value of a referent-slot is updated, then the model predicts both [delete] for the previous value and [new] for the updated value. On the other hand, the [concat] operation is used when the value of triplet needs refinement, such as combining two values, 7 and AM, into single value of 7 AM.
Due to the input length limit of the T5 model, we use the most recent k turns create the previous state and omit the slot descriptions in order to cover more entity-slot-value triplets in the previous state. We get the best results when k = 18 for DIALGEN-AIC and k = 20 for AIC. We present training example for the "AccidentDetails" domain as follows.
Input:
[USER] Oh, sorry about that. You're right, it actually occurred on Mednesday at 11 an. [SYSTEM] Also, I just
wanted to clarify sone information. In our previous conversation, you stated that the accident occurred on records show that actually occurred
Monday
However,
Mednesday at
Can you confirm
which day and time the accident actually occurred? [state] Danage Part [srv] Front Left [rv] Caller [ev] Right I
rv] Global [s] Accident Location [srv] Highway [rv] Global [s] Nun of Passengers [srv] 0 [rv] Global [s]
Mitnesses [srv] Yes [rv] Global [s] Date of Accident [srv] this Monday [rv] Globel [9] Time of Accident [srv]
9:00 an. [rv] Global [s] Subjective Fault [srv] Caller [rv] Caller [domain] AccidentDetaile [possible slots]
Danage Part [s] Accident Location [s] Num of Passengers [s] Witnesses [s] Num of Involved Cars [s] Children
Involved [s] Airbag Deployed [s] Towed [s] Pedestrians Involved [s] Date of Accident [s] Time of Accident [s]
Subjective Fault
Output:
Date of Accident [srv] Mednesday [v] this Monday [vo] [delete] [rv] Global [s] Time of Accident [srv] 11 an. [v] 9:00 an. [vo] [delete] [rv] Global
In the example, the agent (SYSTEM) clarifies the date and time with the caller (USER) because the date and time the caller provides are different from the record in the agent's system. The caller admits the provided time and date are wrong. Therefore, the time and date must be updated. The date previously provided "this Monday" needs to be deleted, so we append an operation [delete] after the value. Similarly, we append the operation after the time "9:00 am."
E DIALGEN
E.1 Data collection cost
The human reviewers were recruited from the university list. They were compensated at rate of $18.69 per hour following our institution's practices. A dialogue, including reviewing synthesizing and annotation processes, required 45-60 minutes, for a final cost per dialogue of $14-19.
E.2 IAA
We follow the methodology in SQuAD (Rajpurkar et al. 2016) for calculating IAA. We select 3 trained workers who participated in data generation as our annotators. They annotated 15% of DIALGEN-AIC. The average time to label a dialogue was 18 minutes. For every dialogue, one annotator is randomly assigned as the reference. We calculate max-Fi of every predicted tuple for every turn and average over all turns, then average across all dialogues.
E.3 AIC ontology
We show the full ontology in Table 10 inncluding domains, slots, and possible values. Possible referents in the AIC ontology: Glohal, Caller, Other Driver, Caller's Passenger, Other Driver's
25

---

Published as a conference paper at COLM 2024
Domain
Posscble Values
Adjuiier
Explain Coverages
[]
Adjaier
Pertidssioon to Record
[yes, nol
Aduater
Set up Inspection
[photo clamm field assignment) Ires, nol
Aduatter
Set up Rental
Cartac Unlo
Finst Nauv
Cintactinlo
Lest Name
Contactialo
Home Addres
Contactindo
Phone Number
Contactinfo
Email Address
Contactinfo
Policy Number
Conbactinfo
Date of Birth
DitverAcloos
Car Molion
lraveling lnrward, bucking, turning, changing linws, stopped, other, unsure)
DetverActions
Spoed
DetverActioes
Distractions
[ocliphane, animals, smoking, passengers, trafiic, cating, not paying attention, other, ursure, no distraction)
DetverActions
Brake
tyes, no. unsure]
DrtverActoons
1omm
[yes, no, unsure]
DriverActions
Tum Signal
[yes, unsure]
DriverActions
Traffic Centrois Cheyed
[yes, no, unwine
Eyieenoes
Police Keport
105, Po, unsure)
Byidences
Police Departmentt Name
Dyidences
Pictures
sceme, after accident, no picture, unsure]
Byidences
Tickets Citations
[caller party cited, other party cited, no party cited, multiple parties cited, unsure no ticket]
Fvidences
Police Report Number
[
Fridences
[yes, unwane]
Detis
Ambullanc
Ies, no, unsure)
InturyDeaails
Body l'rrt Injured
[head. neck, shoulder, chest, abdoenen, back, limb, other]
Detrilleaiils
Injury Type
Penase, brokem fracture. cut sentch, bleeding, stmin spram, sore, other, no iniury]
InjuryDennn
Medical freatment
surpery. CAT scan, hospitaiization
xerny, oter]
tetidentXetalls
Damage Fart
frond, right, back, left, froet right, front left. back left, back right, other, unsure]
MedidenDDetall
Accidcott Location
[posking let, drtreway, highway, roadway, intersection, other]
AecidentDetall Detall
Num Parsengers
(). 2+, unsure]
AccidentDetai Details
Witnesses
[yes, no. unsure]
Accident
Num Involved Cars
[12, 3, smsure]
Arcident Detill
Children Invelved
[yes, unsune]
Arcident
Airbar Deployed
[yes, unwane
AncidentDotill
[yss, unwane
DutidentDMaalls
Ficlestrians Involved
yys,, ureane
AotidentLNtails
AodidentDMtalls
Accident
AetdentDetaills
Subjective
detverl
Carinlo
Modkel
Cartalo
Caztalo
Caztalo
Milsage
Rideshare (Uber/ Lydt
yes, unsure]
TTig
Destination Trip
Trip
Purpose
Ohigin Ttip
Inttrcninirnmment
Weather Visemiity
Clear, clundy, ramny sonny, foggy windy other uneure]
Enviroeentn
Obstraction fions
[yes, urwine]
Enviroment
Condition
(dry, wel, alippery, debris, potholes, strsight, curved, turnel, sleep incline, other unwune]
Envirmment
Signal
bsip sign, silld sign, grewn light, yellow light, light, othere, uroute, no signul sign] tumn lane, shouldes, other, unsurej
Envioment
Description
2, 3, unsure]
Thatichavioommet
Condition
[hoavy, moderate light, other, unsure]
Thaticlavitommeet
Speed
Thaffclivitommeet
(one-waay, twa-way, other, uneure]
Parking Lot Type
[sngled, straight, other, uneure]
Table 10: AIC ontology. Empty lists indicate free-form extractive values.
26

---

Published as a conference paper at COLM 2024
Passenger, and Witness. All referents could be associated with every Nomain//sot,, although in practice certain information is almost always associated with particular referent, egg. Traffic Conditions (heavy, medium, light) always have a Global referent.
E.4 User interface for data collection
We list two main pages of our interface for dialogue generation. They are editing and labeling steps.
First, the editing step Figure 5) page provides dialogue scenarios (slot value pairs), dialogue history, extracted tuples (annotated entity-slot-value triplets), instruction for regeneration, and current subdialogue for editing. A human reviewer provide instruction to guide the LM to generate a desired subdialogue to replace the current subdialogue. If the current subdialogue is satisfied with the reviewer, they can edit turns to fix the minor errors in the subdialogue.
Second, the labeling step page (Figure 6 is an optional page for the DIAL GEN framework. This page is designed for the dialogue state tracking task where the human reviewer can annotate and edit the subdialogue the previous editing step. Note that the labeling step can be fully decoupled from the framework.
The human reviewer will iteratively collaborate with the LM to generate and revise the subdialogues and to annotate the subdialogues until the end of the dialogue.
27

---

Published as a conference paper at COLM 2024
Editing Step
Siot
Redestians
Iraokeed
Mare Mocel
Plist
Sdark
Motion
Car
Roport
Irgury Type Of
Tifffo
Condition
Purposo
Color
Mulic
Squeed
Sko Marks
Ineatrantt
Seedmarkon
Banng Hidtyy
Turn
Party
Tur
Eilo
agent
Thank you for colling Aome! This i Eric, may help you today?
Donected Taploo
Consarint
Deseniction
P-ldensoe
PeayTheats
TrafoTnierree
Instruction Char3PT for moditying some turns regene radng subdialog.
The matrctiee thaight for repenssiigg the cunatt autewig.
sample instrcction
choose sample instructions
You can also type your instruction to guide ChalGPT to enrich the conversation and make more naturall 3
Current Subdialog
For wast roin the ass cervect lecual icomistancy and nemons the redandartinfoomall on the columnn type your woll the berilatt box then bourany tume toid ingenardee now baliing hateos
Modily Some Tums
Regenerate Subdialog
Turne
Party
Your
cany heor Can you please nw come iatamatoo absul Whetl focational
agent
hapeonet edesscttvv syeer
flanndln
how adex
labling the
(Action Label and
(Action
Figure 5: The first step in DIALGEN is create the subdialogue. dialogue scenario table is provided to indicate slots expected to appear in the conversation. A human reviewer selects LM-generated text and edit it as needed. They can also ask the LM to regenerate selected turns or the full subdialogue and optionally provide extra instructions to guide the LM's generation process.
28

---

Published as a conference paper at COLM 2024
Turn labeled
can spon. make you
tupio (domain, value), your oursor scon anmotato one.
what annotaie, check [Link]
(Tum # 14) James (user):
Sure, the other driver seemed going really fast, maybe 45 50 mph There traffic intersection, had the green light entered the intersection. clear weather and there abstructions in my view.
Extracted Tuples in this Turn
calegorisat)
Duplicate Tuples
OtterOhiver Drivenkction, Speed
(Precleed Obwrrriver. DiwerAders, Speed
Keep Concat Updato
Referant Domain
Slot
Value Value
Categorical
Turn
Reterent
Domain
Slot
Value
Value
Categorical
Oteer
DriverActione Speed
45
Drivor
fast
OnwarActonn
Driver
mph.
14
Other
DriverActione Speed
Driver
mph.
indicatio the luplets) tomm the oumant Bart.
Figure 6: A human reviewer selects span and label it. there exists duplicate label, they are prompted to resolve the conflict by selecting to update (as shown), concat, or keep multiple labels.
29

---

Published as a conference paper at COLM 2024
E.5 DIALGEN-AIC dialogues
In Tables
we show sample dialogues from DIALGEN-AIC.
Agent
Thank you for calling DialGen Insurance! This is Greta. How may I help you today?
User
Hello. This is Suzanne. am calling for car accident.
Agent
Alright, Suzanne. First, Lned to verify some information. May plesse have your full name and policy number? Yes, my name is Suzanne Luna and my policy number is 1234567990.
User
Agent
Thank you, Suzanne, And can you please provide me with your phone number and address?
Yes, course. My phone number is (123) 456 7890, and my address 123 Main Street.
A ggnt
Thank you, Suzanne, Now, could you please describe the accident for me?
User
intersection with yield sign. I slowed down and checked for oncoming traffic but didn't oedn'tsee see any cars, so began to turn left. Suddenly, heard loud crunching sound, and my car jerked to stop. I got out of my car assess the damage and saw that black Nissan Maxima had hit the back of my
Certainly, I driving alome my way to the library when approsched
Agent
Thank you for letting me know, Suzanne. Do you know the name the street you were on when the accident happened?
User
Sure, the accident happened at the intersection of Main Street and Broad Avenue.
Agent
And believe you said you were driving alone, that correct?
User
Yes, was alone in my car.
Agent
Were there any witnesses the scene who provided their contact information?
Yes, there were several witmesses
scene who provided their contact information, but didn't get their names.
Agent
De you remember the date and time of the accident?
User
Yes, the accident occurred few nights ago around pm.
Agent
you know
specific day?
User
Let's 's see, it would have been Wednesday no, Thursday night.
Agent
Thank you, Suzanne, Can you tell me if your airhags deployed during the accident?
User
No, the airbags did not deploy.
Agent
Alright, and can describe the traffic conditions at the time of the accident? Was there traffic light at the intersection?
User
Yes,
was a traffic light see. Did you notice anything else about the road or lanes that might have contributed to the accident?
yield sign.
Agent
User
No, nothing else that recall. It all happened fast.
Agent
you remember if you were going at particular speed when the accident happened?
User
No, was not speeding. don't think the other driver was either.
Agent
Alright. Can you tell about damage your sustained? Yes, my car sustained significant damage on the left side.
User
Agent
I'm sorry to hear that. Do you know it police report was filed?
User
Yes, the Houston Police Department look our statements, and report number is 1234567800.
Agent
De you know if either of you received a citation or ticket as result of the accident?
User
Yes, the other driver, Homer Shepherd, received a citation.
Agent
Alright, thank you for letting
know. And can you provide me with make model of your car?
User
Yes, my car is brown sedan, a Toyota Corolla.
Agent
Thank you, Suzanne, And can ask if you sustained any injuries result of the accident?
User
Yes, unfortunately,
sustain some injuries. needed a CT scan hospital after accident.
Agent
Can you tell me where you were injured, Suzanne?
User
My neck and back.
Agent
What was the diagnosis at the hospital?
User
The diagnosis was musele strain and some miner bruising. that, Suzanne. How you feeling now?
Agent
I'm sorry to
User
I'm doing a bit better, thank you for asking.
A Agent
That's good to hear. Now, let me explain your coverage options you.
User
Okay, thank you.
A Agent
with you. You have liability insurance, which covers bodily injuries and property damage others if you at fault in accident, although looks like that's not applicable here. You also have coverage, which pays for damages to your vehicle Okay, got it.
First, let me go over the details of your
event of accident.
User
Agent
Additionaly, you have personal injury protection, which will cover your medical expenses related to injuries sustained in the accident, regardless of who was at fault. However, please note that there may be limits on the amount of coverage provided by this policy.
User
Okay, thank you for explaining that to me.
Agent
Suzanne, can you tell me if your car was towed from the scene of the accident?
User
No, wasn't. was able to drive it home myself.
Agent
Okay,
you letting me know. will just note
report. That should everything the preliminary details.
Is there anything else you'd d like to add, Suzanne?
User
No, that's all, Thank you for your help, Greta.
Agent
You're re welcome.
will process your claim and be touch with you soon about next strps.
User
Okay, thank you so much.
Agent
Have good day, Suzanne.
User
You too, Greta. Bye.
Table 11: Sample DIALGEN-AIC dialogue 1.
30

---

Published as a conference paper at COLM 2024
Agent
Thank you for calling DialGen Insurance! This is Loan. How may I help you today?
User
Hello. This is Joshua. am calling for car accident.
Agent
I'm sorry to hear that you were in an accident. What happened?
User
was last Monday morning
tunnel. There were several cars involved, and my car was hit from the back by a yellow
Dodge Ram. I was injured and had to be taken to the hospital by ambulance.
Agent
Oh my, I'm sorry to hear that. Did you have any passengers in your car?
User
No, I was the only one the car.
Agent
Do you know how many cars were involved total?
User
I think there were about four involved.
Agent
Okay, thank you for that information, Joshua. Can you describe the damage to your car?
User
The back was heavily damaged, and my car undrivable.
Agent
Was your car towed from the scene?
User
Yes, was. Almust all of them had to be.
Agent
Did the police come to the scene of the accident?
User
Yes, they did. They took statements from witnesses, and they also created an accident report that documented all involved
parties' details.
Agent
Great, do you happen to have the police report number and the name of the police department? them right here. The police report number is 12345678, Thank you, Joshua. Was anyone cited or received ticket at the scene?
User
Yes,
was the Philadelphia Police Department.
Agent
User
No, the police sepurtstated stated that no party was cited.
Agent
Okay, thank you letting know. Can you describe the traffic conditions at the time of the accident?
User
Traffic was flowing smoothly in the three-lane road. There was a car that stopped in the lane to my right, and the car behind
them swerved ved into my lane.
Agent
Did you notice any traffic signals or signs that may have cuntributed to the accident?
User
No, there weren't any traffic signals signs at in tunnel.
Agent
see. Can you describe your car's make and model? What year was it made? And what color was it?
User
It's white sedan, a 2018 Honda Accord.
Agent
Thank you for that information, Joshua. Were there any witnesses the accident?
User
Yes, there w were several people who saw the accident happen. Some good Samarilans helped me after the accident and called
Agent
That's 's good to hear. Now, can you tell me about your injuries? What kind of medical treatment did you receive? dislocated my shoulder. They performed CT scan at the hospital to ensure there were no internal injuries. One more thing, Joshua. Can you remind me of the exact date and time of the accident? It was on Monday moming, around 8:30 am.
User
Agent
User
Agent
Okay, just to confirm, that would be the 22nd, correct?
User
Oh, wait. think I may have remembered it wrong. It was actually last Tuesday.
Agent
Thank you for clarifying the date, Jushua. Can you also tell me how fast were you driving when the accident occurred?
User
was driving around 35 miles per hour.
Agent
Thank you for that information, Joshua. Do you have contact information for any the other drivers?
User
Yes, I got Steve Woods' phone number. She was driving the yellow Dodge Ram that hit my car.
Agent
Joshua, can you confirm how the accident occurred from your perspective? I'm a little unclear on some of the details.
User
Sure, was driving the middle lane and noticed stopped car in other lane. The yellow Dodge Ram swerved into my lane to avoid it I couldn't change lanes because the other lane was bumper to bumper. The Dodge Ram hit my back bumper, bounced off, and hit the tunnel wall, damaging the back.
hit the on the right side me. After that, several cars collided with each other. My car spun around
Agent
Thank you for explaining that, Joshua. Just to confirm, were there any traffic cameras at the scene that may have captured the accident?
User
No, I don't believe there were any traffic cameras.
Agent
Thank you for providing all of that information. Joshua. Can ask you to confirm some information about yourself? Sure, go ahuad.
User
Agent
have your full name?
User
Joshua Walters. That's spelled W-A-LL-TE-R-S.
Agent
And can have you provide your phone number and email address please?
User
Yes, my phone number is 555-1234, and my address is 123 Main Street.
Agent
Great, thank you for providing that information. Now it looks like your insurance coverage includes liability insurance, collision coverage, rental car coverage, and gap insurance. Okay.
User
Agent
Do you understand how each those coverages works? Not really, can you explain it to me?
User
Agent
Sure, lability insurance covers any damages or injuries that you may cause to another person or their property. Collision vehicle you're re accident. Rental car coverage pays for rental car while your vehicle is heing repaired. And gap insurance covers the difference between the actual cash value of your car and the balance remaining un your car loan if your car totaled.
coverage pays for damages your
User
Okay, I think I understand now. Thank you for explaining that to me.
Agent
Do you have any other questions for me, Joshua?
User
No, I don't. Thank you for your help.
Agent
Before we end this call, do you happen to have any other details about the accident that you would like to add?
User
No, I think that
everything.
Agent
Okay, thank you for providing all the information, Joshua. We will take care of everything from here.
User
Thank you sa much for your help.
Agent
Of course! And please don't 't hesitate to reach out if have any questions or concerns.
User
I won't. Thanks again.
Table 12: Sample DIALGEN-AIC dialogue 2.
31

---

Published as a conference paper at COLM 2024
Agent
Thank you for calling DialGen Insurance! This William, How may help you today?
User
Hello. This Roberta. I am calling for a car accident.
Agent
am sorry to hear that, Roberta. Before we get started, can 1 get your permission to record this call for quality and training purposes?
User
Yes, you can record the call.
Agent
Thank you. May have your full name and policy number, please?
User
My name is Roberta Oliver, and my policy number is XXXXXXXX
Agent
Got it. Can you please describe the sccident to me?
User
Sure, was driving my gold rideshare car the highway after having dinner restaurant. I was changing lames when Shame Scott's gray Chevrolet Camaro cut me oft, and I had to slam on my brakes to avoid collision. Unfortunately, it was too late, and Shane's 's car crashed into mine.
Agent
Okay, thank you for detailed account. Were there any witnesses who this happen? Any traffic controls?
User
No, I didn't It see any traffic controls around. I'm not sure about any witnesses. Oh, I guess there were the passengers in
Shane's car, but they were loo shaken up to give their stalements to the police.
Agent
Altight. How many passengers were each car?
User
Shane had three passengers in her car. I was alone in mine.
Agent
Thank you for that information, Roberta. Can you provide with the location details of the accident as well as the clate and time occurred?
User
It was May 15th at around 4 in the afternoon. The accident happened on the highway near exit 45B. Thank you for sharing that information, Roberta. forgot to ask carlier, what year is your car?
Agent
User
My car is 2012 model.
Agent
Great, thanks for letting me know. Can you describe the traffic conditions at the time of the accident?
User
It was beautiful day, and the traftic
highway was moving steady pace. There were four and we were both
in the second lane from the left.
Agent
Alright I see. Before proceed further, want let you know that understand how stressful this situation can be, want you to know that am here guide you through the process and make every as clear and easy as possible. How are you feeling?
User
lomestly, I'm foeling pretty overwhelmed right My head has been hurting since the accident, and I'm worried about how much this is going to cost.
Agent
That's perfectly understanuable, Roberta. Just take a deep breath and try to relax. It's good that you're taking steps towards resolving this by calling today. Let's move forward logether, okay? Okay, thank you.
User
Agent
Now you mentioned your head has been hurting since the accident. Did you injure your head during the crash? steering wheel. Since then, I've been having constant headaches, been really difficult focus
Yeah, hit my
on everyday tacks.
Agent
I'm sorry to hear that. Have you seen doctor yet?
User
Yes, went the hospital after accident. They gave a sean which revealed that a minor concussion.
Agent
I'm sorry to hear that. Did they prescribe any treatment or medication?
User
Not really, other than rest and avoiding physical activities. They okayed me to back home immediatelyy but I needed to have my husband check on me every few hours to make sure everything was fine that first night.
Agent
Have you been back to the hospital since to follow up on the headaches?
User
No, but did call my doctor ask her about it. She said that headaches are normal for the first couple of months after concussion, but to go back they get worse.
Agent
see. Thank you for telling me that, Roberta, and I hope the headaches get better soon. Just a few more questions if you'll III accident?
bear with me Can you tell which part of your was damaged
User
The front left side of my car was damaged. The back right side of Shane's as car as well. Thank you for that information. Now understand lerstand that it can be frustrating when there story. However,
Agent
witnesses corroborate your
accident? Perhaps photos of the damage or police report? Yes, the police came to file report. I have a copy of it at home. also took some photos of the damage to my car and Shane's
you have any evidence
User
Agent
Great, that will certainly help. Can you please send those photos over our team? I can provide you with an email address where you can send them. Sure, that would be helpful. What's the email address?
User
Agent
The email is Glales Insurance com. Please put your full name and policy number in the subject line and attach the
photos in the email body.
User
Okay, thanks. will send them over as soon possible. Perfect. there anything clse I can assist you with today, Roberta?
Agent
User
Yes, wondering about the insurance claim process. How long does usually take to get resolution?
Agent
It depends on a few factors, such as the complexity of the case and how much evidence we have. Our team carefully
review your claim and reach out to you within a few business days with a resolution.
User
Okay, that's good know. And what about rental
any other expenses related to the accident?
Agent
We can cerlainly help you out with that you need it. Our besm can sei up rental cars necessary, and we will everything we can to make sure you't re not paying out of pocket for any expenses related to the accident. Will you be needing rental car?
User
No, don't think so.
Agent
Alright, no problem. If you do end up needing a rental car, feel free to let us know. We're here to help in any way we can.
Thanks, appreciate it.
Agent
Of course, Roberta. Is there anything else I can assist you with today?
User
No, that's all for now. Thanks for your help, William.
Agent
was my pleasure, Roberta. Take care and have great day!
User
You too."
Table 13: Sample DIALGEN-AIC dialogue 3.
32

---

Published as a conference paper at COLM 2024
F
Additional analysis
AIC
DialGen-AIC
AIC + DiaKien-AIC
TLB-F
Referent-Slot
50
55
45
50
45
35
40
30
35
Precision
Recall
Precision
Recall
Referent
Slot-value
60
55
55
50
50
45
40
40
35
Precision
Recall
Procision
Recall
Figure 7: TLB and three diagnostic scores for precision and recall (Mg, Mas, and mssv) for the T5-SC model AIC test set.
Figure 7 provides the TLB precision and recall results for the full state updates and different diagnostic scores (referent only, referent-slot, and slot-value). Consistent with the CB results, the biggest benefit of incorporating DIALGEN-AIC is improved recall. While referent, slot, and value all improve, the greatest improvement is in slot values.
G License of artifacts
The license code for (Wolf et al., 2020 is Apache license version 2.0. The license code for Faker and Gender-guesser is MIT and GPLv3 License, respectively. The terms for use of our artifacts will be included in our released package.
33


Reviews:
[{"review_id": "aV5JxIxDHQ", "reviewer": "Reviewer_MWPR", "summary": "This paper evaluates the usefulness of using generated dialogues, with or without the help of human annotators to dynamically correct the generated dialogues, to learn to extract information from dialogues between humans.\nThe paper is very clear. Except for the description of the evaluation measures, CB and TLB, that could be improved.\nThe experiments are well done, and the results are very clear, even if statistical analysis is missing.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our human-corrected synthetic data approach interesting.\n\nRegarding more thorough methods to reduce discrepancy between synthetic and real data: We showed that DialGen is capable of synthesizing dialogue data that is more similar to real data (AIC) than fully automatic data. However, our analysis (section 6.1) also shows there are still gaps, consistent with the reviewer’s comment. We acknowledge that future work can explore additional methods to further minimize the difference between the synthetic and natural data.\n\nRegarding the effect of using real data alongside the generated data with the same amount, the result can be inferred from Figure 4. In the figure, we use 7 real dialogues as a training set (1.3K turns) and obtain a 34.8 TLB score. Using the same amount of generated data should be in the middle of 10% (0.9K) and 20% (1.8K) in the x-axis, suggesting the TLB score of 1.3K-turn generated data will be between 27 - 31. We can add an experiment to confirm the inferred number.\n\nRegarding the missing statistical analysis of the results, we assume the reviewer is suggesting an additional statistical difference test between the experiments. We conducted the statistical difference test on our best model (T5-SC), as shown in Table 3. We used the bootstrap test, recommended in prior research [1][2], to evaluate the difference between the two results using the proposed $CB_4$ F1 metrics. We compared our best T5-SC result (AIC + DIALGEN-AIC) with the T5-SC result using only AIC. The resulting p-value < 0.01 indicates that incorporating both DIALGEN-AIC and AIC as training data is significantly better than using AIC alone.\n\n[1] Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An Empirical Investigation of Statistical Significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995–1005, Jeju Island, Korea. Association for Computational Linguistics.\n\n[2] Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, Australia. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very clear and experiments well made with clear results. The code and generated data will be made available. It is very interesting to see the impact of human correction to generated text used to enrich an existing corpus.", "reasons_to_reject": "Missing statistical analysis of the results.\n\nThe characteristics of generated text, largely different from the real ones, could be explored more thoroughly and methods to reduce this discrepancies could be studied.", "questions_to_authors": "What would be the effect of adding a certain amount of human data compared to adding the same amount of generated data? This could be studied in keeping a part of the original data apart for later use. But as the original data is quite small, this could be difficult.", "extra_scores": {}}, {"review_id": "R1VyJvhJkb", "reviewer": "Reviewer_PnLs", "summary": "To palliate the lack of available training data in the context of task-oriented-dialogue, the paper investigates the addition of a human in the loop, when performing data-augmentation with Large Language Models for a task of extracting information from auto insurance call centre dialogues (AIC corpus), using public (anonymized) synthetic data to improve performance on the private dataset.\nThe experiment where done with an Auto Insurance Claim information collection task, where the\ncurrently active slot-value set represents the dialogue state.\nThe authors observed a 25% relative performance increase (F1 measure on information extraction\nfrom dialogue history) when augmenting a small set of conversations with synthetic data which were generated by prompting a LLM and then checked by a human. During the generation,\nthe human checks and if needed edit the subdialogue generated and/or the prompt before the generating the next subdialogue in an iterative process tacking into account the full dialogue history,\nuntil completion of the dialogue.\nThe authors propose a contribution in four parts: a collaborative human-LLM framework,\na custom AIC dataset of conversation with annotations (slot values with entity linking and multiple\nslot value tracking), a new entity-centered scoring methodology based on the previous contribution\nand finally experiments with LLMs. Comparative experiments against a fully automatic approach have shown the benefit of human collaboration both for dialogue generation and annotation. The author found that synthetic data has fewer turns, longer turns, and less variance in length, while fully automatic data is the most extreme and the human intervention results in much longer and more varied dialogues, which are still from reaching the complexity of human-human dialogues.\nIn the experiments, the authors deployed first in-context learning with Chat-GPT-3.5-0301 (the same model as used for the data augmentation with a human-in-the-loop) and SBERT as a retriever to predict slot-value pairs (state changes) and the corresponding entity (referent) for the pair with two different prompting scenarios.\nSecond the ran a series of experiments by fine-tuning different version of the T5 and T5-long transformer models. Third, they investigated replacing the human in the loop with ChatGPT and GPT-4 (keeping T5 model for finetuning with the entirely synthetic data) and showed that despite better performance with GPT-4 than chatGPT, the synthetic data produced with a human collaboration provided better quality training data than the fully automatic approach.\n\nThe paper addresses a hot topic nowadays about the extend to which a large language models could possibly be used \nfor generating realistic language data. It makes a significant contribution based on state of the art models that shows that\nif LLMs can help augment training data, they are far from making human intervention obsolete for reaching human performance level\nas soon as the task becomes complex.\nNote that IMHO the fact that the authors used ChatGPT-3 and GPT4, which are not open model and thus limit their use\nin a scientific experiment (because of the resulting limitation it poses for reproducibility) is mitigated by the fact that these models \nhave been used to produce a relatively limited amount of data and that the contribution is about the methodology that could use\nother open large language model. Nevertheless, it would have been interesting to have a test done with such model for comparison.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding that our methodology is impactful in contributing to the discussion around language model (LM) performance and showing that the current LM is still limited without human correction in data synthesis and annotation. We acknowledge that using open-source LMs would increase the reproducibility and enhance our findings. We used these private models as of the paper submission because they are the current state-of-the-art models in this field. Recent open LMs, such as LLaMA3 (released after this work), could provide an alternative LM for our human-in-the-loop framework. We leave it as a future work to explore.\n\nWe acknowledge the raised issue of reproducibility of using LMs. Based on recommendations from OpenAI, the temperature value ranges from 0 to 2, with lower values indicating greater determinism and higher values indicating more randomness.  In our quantitative experiments, only the IC-DST experiment involves a private LM. To enhance the reproducibility of the IC-DST experiment on the publicly available DialGen-AIC, we set the temperature to 0 and report the API version (e.g., gpt-3.5-turbo-0301).\n\nThank you for your suggestion about the footnote. We will edit the paper and make the footnote appear at the first-mentioned position in the main content. Regarding the readability suggestion about the version of API calls of language models, we added a table to indicate the version of different LMs in Appendix C.2 of page 16 due to the page limit. We will add the SBERT and T5 into the table and add a reminder to inform readers the table is in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper makes a significant contribution on a hot research topic with state of the technology and\nreport an interesting result, that shows that Large Language Models are far from reaching human performance\nfor data augmentation for a relatively complex language task.\n\nThe experiments are clearly reported, in details with language examples in annex and following best practice of the domain and the state of the art is up to date.", "reasons_to_reject": "The fact that the experiments rely partially on the use of Large Language Models which are not open, raising\nquestions about reproducibility of the results, not forgetting the fact that the original source corpus\nis private data.", "questions_to_authors": "page 6, section 4,\n\"we apply D IAL G EN with ChatGPT as the LM backbone to create DIALGEN -AIC,\"\nThe version  of ChatGPT that has been used is given later on top of page 7 (with a complement of\ninformation in a footnote), it would be better to have the information with the first occurrence\nof the term. Maybe having a table or synoptic diagram in the annexe recapitulating in which experiments\nthe various models (ChatGPT, SBERT, T5, GPT4) have been used would be helpful to the reader.", "extra_scores": {}}, {"review_id": "Doqq7JvrOa", "reviewer": "Reviewer_wSD9", "summary": "The paper presents a novel approach in the task-oriented dialogue construction. Authors propose human-in-the-loop scheme and show that it helps to improve the final quality on the private dataset. A number of settings, including fully automatic dialogue generation are tested, and new dataset is created.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our work impactful in contributing to the research field.\n\nRegarding reproducing results on a public dataset, we acknowledge that the additional results on a public dataset would increase the reproducibility of this work. However, assessing impact on real, complex, goal-oriented conversations is a key objective of our work, and it is challenging to find a public dataset that contains realistic dialogues due to the privacy issue.\n\nWe follow previous work (IC-DST) to represent the ontology via a SQL table format. In Appendix D3, we show an example of the prompt. For example, the domain Adjuster (as shown in the following text block) contains the domain slots Explain Coverages, Permission to Record, Set up Inspection, Set up Rental. The domain slots with CHECK values indicate they are categorical slots. Otherwise, the slots are non-categorical. We will update our description to clarify the ontology representation format.\n\n```\nCREATE TABLE Adjuster(\n'Explain Coverages' TEXT,\n'Permission to Record' TEXT CHECK ('Permission to Record' IN 'Yes', 'No'),\n'Set up Inspection' TEXT CHECK ('Set up Inspection' IN 'Quick Photo Claim', 'Field\nAssignment'),\n'Set up Rental' TEXT CHECK ('Set up Rental' IN 'Yes', 'No'),\n)\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Important research field combining solid practical and theoretical results.\n\nSeveral important contributions, including novel dataset and model.\n\nDetailed explanation of the approach, that makes it highly reproducible.\n\nFine-grained and impressive evaluation supporting major contributions of the paper.", "reasons_to_reject": "It would be nice to reproduce results on the open dataset or at least provide some preliminary evaluation.\nThe paper is not clear in some minor moments. For instance, I did not get the description of the ontology representation format. SQL tables are mentioned but without further explanation.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pRiJFgPoHG", "reviewer": "Reviewer_fNg6", "summary": "The contributions on pg 2 are an accurate summary:   (1) the paper describes DialGen,  a collaborative human-LM framework for generating synthetic dialogues in domains where privacy constraints prevent release of data;  (2) DialGen-AIC dataset for real-world auto insurance call centre data;  (3) an entity-based scoring methodology; (4) results showing improvements from human collaboration over a fully automatic LM-based approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our human-LM collaborative framework interesting. \n\nWe proposed a new IE metric because the DST metrics, joint goal accuracy (JGA) and slot accuracy, can not well reflect the model performance on our task. Here are more details. \n\nFirst, JGA is chosen because DST tasks require database queries; hence accurate prediction is needed. Queries are issued after most turns, so evaluating average performance at all turns makes sense. In contrast, in our IE task, where information is accumulated (and corrected) for a final summary. Our goal in evaluating car accident claims is to assess the overall extraction quality. Minor errors (e.g., additional word in non-categorical slots) should not significantly impact the readability of the extracted information. Therefore, we chose more flexible F1 metrics. \n\nSecond, both slot accuracy and our IE metric exhibit the same trend where later turns receive lower scores. It is because the later turns are harder to predict because more information is presented in a dialogue. Slot accuracy is computed at each turn and defined as:\n\n$\\text{Slot accuracy} = \\frac{T - M - W}{T}$,\n\nT: # all predefined slots.\nM: # missed slots that the model does not predict among the gold slots.\nW: # wrongly predicted slots that do not exist in the gold slots.\n\nEarly turns usually contain a few gold slots, resulting in high scores for every model and making the performance difference between models become trivial. In contrast, our IE metric evaluates at specific dialogue points (quarter, half, three-quarters, end), mitigating the issue of misevaluation of models at earlier turns.\n\nRegarding the use of powerful models, we chose finetuned T5 models for two reasons: (1) previous work [1] shows that it outperforms large decoder-only models in finetuned domains, and (2) our resources for finetuning on private data are limited, preventing the use of larger models.\n\nMoreover, in this work, we explored the experiments with the in-context learning (IC-DST w/ ChatGPT) on the synthesized data due to the privacy concern. We find that IC-DST performs worse than a finetuned T5 up to 12% as shown in Table 2. However, we acknowledge that it could be interesting to explore recent open models (e.g. LLaMA3) on the private data in the future if resources permit.\n\n[1] E. Lehman, E. Hernandez, D. Mahajan, J. Wulff, M. J. Smith, Z. Ziegler, D. Nadler, P. Szolovits, A. Johnson, and E. Alsentzer. Do we still need clinical language models?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "An interesting approach to human-LM collaboration for dialogue generation.   Including humans-in-the-loop yields lengthier and more complex dialogues than LM only approaches, suggesting reasonable ways to include humans in dialogue generation as well as possible opportunities for improvements in LM-only approaches.", "reasons_to_reject": "No strong reasons to reject, except that the paper may be of interest to a relatively narrow audience.   The general objectives are likely to be broadly interesting ('  a human-LM collaborative framework for dialogue generation (DIALGEN) that leverages the scalability and creativity of generative models, yet retains controllability through humans'),  but the specifics ('extracting information from auto insurance call center dialogues, using public synthetic data to improve performance on a private dataset') are fairly specific, although interesting for dialogue researchers.", "questions_to_authors": "This is a minor point, but I'm curious about the motivation to adopt IE metrics in place of DST metrics.   in Section 3.3, there is the assertion 'Directly adopting DST metrics for dialogue-based IE is not ideal, because they overemphasize earlier parts of a conversation and do not disentangle the effects of error propagation across turns.'   Is a similar effect present in the F1 scores in Table 2?  It seems that IE metrics are also higher early in the dialogue.\n\nResults (e.g. Table 4) show that data produced by DialGen w/ human annotators can be used to improve T5 models.   What about uses of the data to improve more powerful models or models that can't practically be fine-tuned?  These are likely to be of interest.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces DialGen, a human-in-the-loop framework for generating synthetic dialogues to address the lack of training data for information extraction (IE) tasks in privacy-sensitive domains. The authors demonstrate a 25% relative improvement in IE performance by augmenting real call center conversations with synthetic data generated and refined through human-LM collaboration.

Reviewer Consensus:

* The paper addresses a significant and relevant problem, given the privacy constraints in real-world dialogue data.
* The paper is clear, well-structured, and adheres to best practices in the field.
* The inclusion of humans in the loop for generating and refining synthetic dialogues is valuable and leads to improved performance.

Reviewer Disagreements:

* One reviewer noted the absence of statistical analysis, which the authors addressed by providing additional tests.
* Concerns were raised about the generalizability of the findings due to the reliance on a single private dataset and non-open LMs. The authors acknowledged this limitation but highlighted the focus on real-world, complex dialogues.
* A suggestion was made to further analyze the discrepancies between synthetic and real data, which the authors agreed with.

Overall Assessment and Recommendation:

Despite minor disagreements, the paper receives positive reviews for its contribution to the field. The proposed framework and dataset are valuable resources for future research. The authors' detailed rebuttal effectively addresses concerns.

Based on the overall positive reviews, I recommend accepting this paper. It offers valuable insights into the potential of human-LM collaboration for data generation and annotation and its implications for improving IE performance in privacy-sensitive domains.
Decision: Accept