SlowGuess/ACL_2020_OCR huggingface dataset:
Giovanni Campagna Agata Forsciarz Mehrad Moradshahi Monica S. Lam
{gcampaign,agataf,mehrad,lam}@cs.stanford.edu
Zero-shot transfer learning for multi-domain dialogue state tracking can allow us to handle new domains without incurring the high cost of data acquisition. This paper proposes new zero-short transfer learning technique for dialogue state tracking where the in-domain training data are all synthesized from an abstract dialogue model and the ontology of the domain. We show that data augmentation through synthesized data can improve the accuracy of zero-shot learning for both the TRADE model and the BERT-based SUMBT model on the MultiWOZ 2.1 dataset. We show training with only synthesized in-domain data on the SUMBT model can reach about  $2/3$  of the accuracy obtained with the full training dataset. We improve the zero-shot learning state of the art on average across domains by  $21\%$ .
Automated conversational agents can reduce the costs of customer support, a necessary service in just about every business. However, training a goal-directed dialogue agent for a domain often requires acquiring annotated dialogues to cover all possible conversation flows. Commonly, this is done using the Wizard-of-Oz technique (Kelley, 1984), where two crowdsourced workers converse with each other, while also annotating the state at each turn. This technique has been employed to construct several datasets (Hemphill et al., 1990; Wen et al., 2017; Yu et al., 2019). Recently, it has been used to build the MultiWOZ dataset (Budzianowski et al., 2018), a large corpus of dialogues across 7 domains.
Unfortunately, not only is the initial acquisition expensive, annotating dialogues correctly has proven to be challenging due to human errors, delays in annotation, inconsistent conventions, and normalization issues (Eric et al., 2019; Zhou and
S: START
U: Can you help with information regarding a food place? I need to book at 15:45.
S: SEARCHREQUEST restaurant(book_time = "15:45")
A: How about the restaurant with name La Tasca and Italian food?
U: Can you find something which serves seafood?
S: SEARCHREQUEST restaurant(book_time = "15:45", food = "seafood")
A: What date are you looking for?
U: Thursday please.
S: SEARCHREQUEST restaurant(book_time = "15:45", food = "seafood", book_day = "thursday")
A: How about the Copper Kettle? It is a food place with seafood food.
U: What is the price range and the area?
S: SLOTQUESTION restaurant(book_time = "15:45", food = "seafood", book_day = "thursday", price_range =?, area =?)
A: The Copper Kettle is a moderately priced restaurant in the north of the city. Would you like a reservation?
U: No, thanks.
S: CLOSE restaurant(book_time = "15:45", food = "seafood", book_day = "thursday")
A: Can I help with you anything else?
U: Thank you, that will be it for now.
S: END restaurant(book_time = "15:45", food = "seafood", book_day = "thursday")
Figure 1: An example of a dialogue that can be synthesized from our templates. 'U:' indicates the user, 'A:' the agent, and 'S:' is the dialogue state at each turn.
Small, 2019). The MultiWOZ dataset still has significant inconsistencies (Zhou and Small, 2019) despite having been constructed through multiple rounds of annotations (Budzianowski et al., 2018; Eric et al., 2019).
We observe empirically from the MultiWOZ training data that conversations in all the domains follow the same pattern: the agent and user start by greeting each other, tehn they converse to find a proposal that satisfies the user, the user provides additional required information, and finally the agent completes the user's transaction.
To facilitate transfer learning, we create an abstract model of dialogues that is independent of the domain of the conversation. In this paper we will
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 122-132 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
focus on dialogues for transactions; other kinds of dialogues such as opinion sharing will have different models. We have developed an algorithm that accepts an ontology of a domain and a few phrases commonly used in that domain. The algorithm synthesizes dialogue training data based on an abstract dialogue model. The dialogue synthesized consists of turns of conversation, each of which has a start state, an agent utterance, a user utterance, and an end state. The start and end states summarize the semantics of the conversation at those points. An example of a dialogue that can be synthesized by our model is shown in Fig. 1.
To transfer knowledge to a new domain in a zero-shot setting, we train with the synthesized data for the new domain together with existing data for other domains. In addition, we adapt training samples from related domains by substituting them with the vocabulary of the new domain. We can improve the accuracy of the abstract dialogue model as well as the state-tracking neural network by iteratively refining the model based on the error analysis on the validation data, and by introducing additional annotations in the new domain. Note that the abstract dialogue model can be also used directly to implement the agent itself.
- A new zero-short transfer learning technique for dialogue state tracking where the indomain training data are all synthesized from an abstract dialogue model and the ontology of the domain.
- Our approach improves over the previous state-of-the-art result on zero-shot transfer learning for MultiWOZ 2.1 tasks by  $21\%$  on average across domains.
- We show that our approach improves the accuracy for TRADE (Wu et al., 2019), an RNN-based model, and SUMBT (Lee et al., 2019), a BERT-based model (Devlin et al., 2019), suggesting that our technique is independent of the specific model used.
- Our experimental results show that synthesized data complements BERT pretraining. The BERT-based SUMBT model can, in a purely zero-shot fashion, achieve between  $61\%$  and  $92\%$  of the accuracy obtained by a model trained on the full dataset. We propose combining pretrained models with synthesized data as a general technique to bootstrap new dialogue state trackers.
Dialogue Datasets and Synthesis. Synthesized data (in training and evaluation) was proposed by Weston et al. (2015) to evaluate the ability of neural models to reason compositionally, and was also used in visual question answering (Johnson et al., 2017a; Hudson and Manning, 2019) and semantic parsing (Lake and Baroni, 2018).
Wang et al. (2015) proposed synthesizing data, then crowdsourcing paraphrases to train semantic parsers. Various semantic parsing datasets have been generated with this technique (Su et al., 2017; Zhong et al., 2017) and the technique has also been adapted to the multiturn setting (Cheng et al., 2018; Shah et al., 2018). While it tends to be well-annotated, paraphrase data is expensive to acquire, and these datasets are very small.
More recently, we proposed training with both a large amount of synthesized data and a small amount of paraphrase data for semantic parsing of single sentences (Campagna et al., 2019; Xu et al., 2020). We showed that training with such data can perform well on real-world evaluations. This paper extends this work to the multi-turn setting. Dialogues are more complex as they need to capture information, such as the abstract dialogue state, that is not present in the target annotation (domain and slot values). We extend the synthesis algorithm to operate based on a dialogue model, tracking enough information to continue the dialogue. We also present a novel dialogue model that is suitable for synthesis.
Dialogue State Tracking. Dialogue state tracking is a long-studied field, starting with the first Dialogue State Tracking Challenge (Williams et al., 2014). A review of prior work can be found by Williams et al. (2016).
Previous works on DST use different approaches, ranging from using handcrafted features to elicit utterance information (Henderson et al., 2014; Wang and Lemon, 2013). Mrkšić et al. (2017) use Convolutional Neural Networks to learn utterance representations. However, their models do not scale as they do not share parameters across different slots. Zhong et al. (2018) and Nouri and Hosseini-Asl (2018) propose a new global module that shares information to facilitate knowledge transfer. However, they rely on a predefined ontology. Xu and Hu (2018) use a pointer network with a Seq2Seq architecture to
handle unseen slot values. Lee et al. (2019) use a pre-trained BERT model (Devlin et al., 2019) to encode slots and utterances and uses multi-head attention (Vaswani et al., 2017) to find relevant information in the dialogue context for predicting slot values. Wu et al. (2019) introduce an encoder-decoder architecture with a copy mechanism, sharing all model parameters between all domains. Zhou and Small (2019) formulate multidomain DST as a question answering task and use reading comprehension techniques to generate the answers by either span or value prediction.
Johnson et al. (2017b) propose single encoder-decoder models for zero-shot machine translation by encoding language and input sentence jointly, and Zhao and Eskenazi (2018) propose cross-domain zero-shot language generation using a cross-domain embedding space.
Modelling of Dialogues. Previous work already proposed general models of dialogues as finite state machines (Jurafsky et al., 1997; Bunt et al., 2017; Yu and Yu, 2019). Existing models are optimized for analyzing existing human conversations. Our dialogue model is the first suitable for synthesis, carrying enough information to continue the dialogue.
Gupta et al. (2018) previously proposed a different annotation scheme for dialogues, using a hierarchical representation scheme, instead of the more typical intent and slot. Their work is complementary to ours: our method of dialogue synthesis is applicable to any annotation scheme. In this paper, we focus on the existing annotation scheme used by the MultiWOZ dataset.
3 Dialogue-Model Based Synthesis
In this section, we first define abstract dialogue models, then describe how we can generate dialogues based on the model. We also describe the techniques we use to adapt training dialogues from other domains to the new domain.
3.1 Abstract Dialogue Model
We define a dialogue model with finite sets of abstract states, agent dialogue acts, user dialogue acts, and transitions, defined below. The abstract dialogue for transactions we use in this paper is shown in Table 1.
The abstract states capture the typical flow of a conversation in that model, regardless of
the domain. For example, a transaction dialogue model has states GREET, SEARCHREQUEST, COMPLETEREQUEST, COMPLETETRANSACTION, and CLOSECONVERSATION, etc. Each domain has a set of slots; each slot can be assigned a value of the right type, a special DONTCARE marker indicating that the user has no preference, or a special “?” marker indicating the user is requesting information about that slot. Thus, we can summarize the content discussed up to any point of a conversation with a concrete state, consisting of an abstract state, and all the slot-value pairs mentioned up to that point. Where it is not ambiguous, we refer to the concrete state as the state for simplicity.
All possible agent utterances in a dialogue model are classified into a finite set of agent dialogue acts, and similarly, all the possible user utterances into a finite set of user dialogue acts. Examples of the former are GREETUSER, ASKQUESTION, ANSWER, OFFERRESERVATION; examples of the latter are ASKBYNAME, ADDCONSTRAINTS, ACCEPT, REJECT.
Each transition in the model describes an allowed turn in a dialogue. A transition consists of an abstract start state, an agent dialogue act, a user dialogue act, and an abstract end state.
3.2 Dialogues from an Abstract Model
A dialogue is a sequence of turns, each of which consists of a start state, an agent utterance, a user utterance, and an end state. We say that a dialogue belongs to a model, if and only if,
1. for every turn, the start state's abstract state, the dialogue act of the agent utterance, the dialogue act of the user utterance, and the end state's abstract state constitute an allowed transition in the model.
2. the slot-value pairs of each end state are derived by applying the semantics of the agent and user utterances to the start state.
3. the first turn starts with the special START state, and every turn's end state is the start state of the next turn, except for the last turn, where the end state is the special END state.
3.3 Synthesizing a Turn with Templates
We use templates to synthesize dialogues in a domain from an abstract dialogue model and a domain ontology. In this paper, we introduce dialogue model templates which specify with grammar rules how to generate a turn of a dialogue from
From Abstract State	Agent Dialogue Act	User Dialogue Act	To Abstract State
Start	Greet	Greeting
Ask by name	Info request
Ask with constraints	Search request
Greet	Greet	Ask by name	Info request
Ask with constraints	Search request
Search request	Ask to refine search	Provide constraints	Search request
Ask question	Answer question	Search request
Propose constraint	Accept constraint	Search request
Add constraints	Search request
Propose entity	Accept	Complete request
Add constraints	Search request
Reject	Search request
Ask slot question	Slot question
Ask info question	Info question
Empty search, offer change	Change constraints	Search request
Insist	Insist
Info request	Provide info, offer reservation	Accept	Accept
Provide reservation info	Accept
Ask info question	Info question
Info question	Answer, offer reservation	Accept	Accept
Provide reservation info	Accept
Thanks	Close conversation
Slot question	Answer, offer reservation	Accept	Accept
Add constraint	Search request
Insist	Repeat empty search	Apologize	Close conversation
Change constraints	Search request
Complete request	Offer reservation	Accept	Accept
Thanks	Close conversation
Accept	Ask missing slots	Answer question	Complete transaction
Complete transaction	Execute	Ask transaction info	Transaction info question
Thanks	Close conversation
Error	Thanks	Close conversation
Transaction info question	Answer	Thanks	Close conversation
Close conversation	Anything else	Thanks	End
Table 1: Our abstract dialogue model for transaction dialogues. Each row represents one transition between abstract dialogue states.
a transition in the abstract model. They create possible agent and user utterances matching the agent and user dialogue acts in the transition, and they include a semantic function to ensure the utterances make sense given the input state. For example, the user should ask about a slot only if its value is not already known. The semantic function returns an output state that matches the semantics of the utterances. The slot values of the output state are used as the annotation for the turn when training the dialogue state tracker.
As an example, the SLOTQUESTION template shown in Fig. 2 corresponds to the 13th transition in the dialogue model in Table 1. The following agent and user utterances, separated by a delimiting token  $<\mathrm{sep}>$ , are examples of dialogue acts PROPOSENTY and ASKSLOTQUESTION. They transition the abstract state SEARCHREQUEST to the abstract state SLOTQUESTION.
State: SEARCHREQUEST restaurant(..)
Agent: How about Curry Garden? It is an Indian restaurant in the south of town. <sep>
User: Is it expensive?
State: SLOTQUESTION restaurant(..., price = "?")
In this case, the non-terminals NAME, NP, ADJ_SLOT are expanded into domain-dependent phrases "Curry Garden", "Indian restaurant in the south of town", and "expensive", respectively, and the results of their semantic functions, name, np, adj_slot, are (sets of) slot-value pairs: name = "Curry Garden"; {food = "Indian", area = "south"}; price = "expensive". The semantic function of SLOTQUESTION checks that the input state does not already include a value for the price slot, and the price is not mentioned by the agent at this turn. It returns, as the new state, the old state with a "?" on the price.
All the non-dialogue specific templates are introduced by Xu et al. (2020). We have extended this template library, originally intended for database queries, to return slot-value pairs as semantic function results. Readers are referred to Xu et al. (2020) for details. This library has
Figure 2: The SLOTQUESTION template and other nondialogue specific templates used to generate the example interaction.
four kinds of domain templates. Domain Subject Templates describe different noun phrases for identifying the domain. Slot Name Templates describe ways to refer to a slot name without a value, such as "cuisine", "number of people" or "arrival time". Slot Value Templates describe phrases that refer to a slot and its value; they can be a noun phrase ("restaurants with Italian food"), passive verb phrase ("restaurants called Alimentum"), active verb phrase ("restaurants that serve Italian food"), adjective-phrase ("Italian restaurants"), preposition clauses ("reservations for 3 people"). Finally, Information Utterance Templates describe full sentences providing information, such as "I need free parking", or "I want to arrive in London at 17:00". These are domain-specific because they use a domain-specific construction ("free parking") or verb ("arrive").
Developers using our methodology are expected to provide domain templates, by deriving them manually from observations of a small number of in-domain human conversations, such as those used for the validation set.
3.4 Synthesizing a Dialogue
As there is an exponential number of possible dialogues, we generate dialogues with a randomized search algorithm. We sample all possible transitions uniformly to maximize variety and coverage. Our iterative algorithm maintains a fixed-size working set of incomplete dialogues and their current states, starting with the empty dialogue in the START state. At each turn, it computes a random
sample of all possible transitions out of the abstract states in the working set. A fixed number of transitions are then chosen, their templates expanded and semantic functions invoked to produce the new concrete states. Extended dialogues become the working set for the next iteration; unextended ones are added to the set of generated results. The algorithm proceeds for a maximum number of turns or until the working set is empty.
The algorithm produces full well-formed dialogues, together with their annotations. The annotated dialogues can be used to train any standard dialogue state tracker.
3.5 Training Data Adaptations
We also synthesize new training data by adapting dialogues from domains with similar slots. For example, both restaurants and hotels have locations, so we can adapt a sentence like "find me a restaurant in the city center" to "find me a hotel in the city center". We substitute a matching domain noun phrase with the one for the new domain, and its slot values to those from the target ontology.
We also generate new multi-domain dialogues from existing ones. We use heuristics to identify the point where the domain switches and we concatenate single-domain portions to form a multidomain dialogue.
4.1 The MultiWOZ Dataset
The MultiWOZ dataset (Budzianowski et al., 2018; Eric et al., 2019) is a multi-domain fully-labeled corpus of human-human written conversations. Its ontology has 35 slots in total from 7 domains. Each dialogue consists of a goal, multiple user and agent utterances, and annotations in terms of slot values at every turn. The dataset is created through crowdsourcing and has 3,406 single-domain and 7,032 multi-domain dialogues.
Of the 7 domains, only 5 have correct annotations and any data in the validation or test sets. Following Wu et al. (2019) we only focus on these 5 domains in this paper. The characteristics of the domains are shown in Table 2.
4.2 Machine Learning Models
We evaluate our data synthesis technique on two state-of-the-art models for the MultiWOZ dialogue state tracking task, TRADE (Wu et al., 2019) and SUMBT (Lee et al., 2019). Here we
Attraction	Hotel	Restaurant	Taxi	Train
# user slots	3	10	7	4	6
# agent slots	5	4	4	2	2
# slot values	167	143	374	766	350
# real dialogues	3,469	4,196	4,836	1,919	3,903
# in-domain turns	10,549	18,330	18,801	5,962	16,081
# in-domain tokens	312,569	572,955	547,605	179,874	451,521
# domain subject templates	3	5	4	2	4
# slot name templates	15	17	21	18	16
# slot value templates	7	30	30	37	42
# information utterance templates	1	14	13	13	27
# synthesized dialogues	6,636	13,300	9,901	6,771	14,092
# synthesized turns	30,274	62,950	46,062	35,745	60,236
# synthesized tokens	548,822	1,311,789	965,219	864,204	1,405,201
transfer domain	Restaurant	Restaurant	Hotel	Train	Taxi
overlapping slots	2	6	6	4	4
Table 2: Characteristics of the MultiWOZ ontology, the MultiWOZ dataset, the template library, and the synthesized datasets for the zero-shot experiment on the 5 MultiWOZ domains. "user slots" refers to the slots the user can provide and the model must track, while "agent slots" refer to slots that the user requests from the agent (such as the phone number or the address). Note that total number of dialogues is smaller than the sum of dialogues in each domain due to multi-domain dialogues.
give a brief overview of each model; further details are provided in the respective papers.
TRADE TRAnsferable Dialogue statE generator (TRADE) uses a soft copy mechanism to either copy slot-values from utterance pairs or generate them using an Recurrent Neural Network (RNN) (Sutskever et al., 2014) decoder. This model can produce slot-values not encountered during training. The model is comprised of three main parts: an RNN utterance encoder which generates a context vector based on the previous turns of the dialogue; a slot-gate predictor indicating which (domain, slot) pairs need to be tracked, and a state generator that produces the final word distribution at each decoder time-step.
SUMBT Slot-Utterance Matching Belief Tracker (SUMBT) uses an attention mechanism over user-agent utterances at each turn to extract the slot-value information. It deploys a distance-based non-parametric classifier to generate the probability distribution of a slot-value and minimizes the log-likelihood of these values for all slot-types and dialogue turns. Specifically, their model includes four main parts: the BERT (Devlin et al., 2019) language model which encodes slot names, slot values, and utterance pairs, a multi-head attention module that computes an attention vector between slot and utterance representations, a RNN state tracking module, and a discriminative classifier which computes
the probability of each slot value. The use of similarity to find relevant slot values makes the model depend on the ontology. Thus the model is unable to track unknown slot values.
4.3 Software and Hyperparameters
We used the Genie tool (Campagna et al., 2019) to synthesize our datasets. We incorporated our dialogue model and template library into a new version of the tool. The exact version of the tool used for the experiment, as well as the generated datasets, are available on GitHub<sup>1</sup>. For each experiment, we tuned the Genie hyperparameters separately on the validation set.
For the models, we use the code that was released by the respective authors, with their recommend hyperparameters. For consistency, we use the same data preprocessing to train both TRADE and SUMBT.
5.1 Data synthesis
Our abstract transaction dialogue model has 13 abstract states, 15 agent dialogue acts, 17 user dialogue acts, and 34 transitions (Table 1). We have created 91 dialogue templates for this model. Dialogue templates were optimized using the validation data in the "Restaurant" domain.
<https://github.com/stanford-oval/ zero-shot-multiwoz-acl2020
Model	Synth.	Joint	Slot Acc.
TRADE	no	44.2	96.5
yes	43.0	96.4
SUMBT	no	46.7	96.7
yes	46.9	96.6
Table 3: Accuracy on the full MultiWOZ 2.1 dataset (test set), with and without synthesized data.