CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers
Authors: SHIYANG LI
Semih Yavuz
Kazuma Hashimoto
Jia Li
Tong Niu
Nazneen Rajani
Xifeng Yan
Yingbo Zhou
Caiming Xiong
Venue: ICLR 2021 Poster (2021)
Keywords: task-oriented dialogue
dialogue state tracking
robustness
dst
evaluation

Abstract: Dialogue state trackers have made significant progress on benchmark datasets, but their generalization capability to novel and realistic scenarios beyond the held- out conversations is less understood. We propose controllable counterfactuals (COCO) to bridge this gap and evaluate dialogue state tracking (DST) models on novel scenarios, i.e., would the system successfully tackle the request if the user responded differently but still consistently with the dialogue flow? COCO leverages turn-level belief states as counterfactual conditionals to produce novel conversation scenarios in two steps: (i) counterfactual goal generation at turn- level by dropping and adding slots followed by replacing slot values, (ii) counterfactual conversation generation that is conditioned on (i) and consistent with the dialogue flow. Evaluating state-of-the-art DST models on MultiWOZ dataset with COCO-generated counterfactuals results in a significant performance drop of up to 30.8% (from 49.4% to 18.6%) in absolute joint goal accuracy. In comparison, widely used techniques like paraphrasing only affect the accuracy by at most 2%. Human evaluations show that COCO-generated conversations perfectly reflect the underlying user goal with more than 95% accuracy and are as human-like as the original conversations, further strengthening its reliability and promise to be adopted as part of the robustness evaluation of DST models.

Published as a conference paper at ICLR 2021
CoCo: CONTROLLABLE COUNTERFACTUALS FOR EVALUATING DIALOGUE STATE TRACKERS
Shiyang Li Semih Yavuz,
Kazuma Hashimoto, Jia Li, Tong Niu
Nazneen Rajani, Xifeng Yan Yingbo Zhou, Caiming Xiong Salesforce Research University of California, Santa Barbara fsyavuz, k.hashimoto, jia.li, tniu, nazneen. ra rajani, yingbo. zhou, beiong]@saleefoocee {shiyangii, xyan]@ucsb.c edu
ABSTRACT
Dialogue state trackers have made significant progress on benchmark datasets, but their generalization capability to novel and realistic scenarios beyond the held- out conversations less understood. We propose controllable counterfactuals (CoCo) to bridge this gap and evaluate dialogue state tracking (DST) models on novel scenarios, i... would the system successfully tackle the request if the user responded differently but still consistently with the dialogue flow? CoCo leverages turn-level belief states as counterfactual conditionals to produce novel conversation scenarios in steps: (i) counterfactual goal generation at turn- level by dropping and adding slots followed by replacing slot values, (ii) coun- terfactual conversation generation that is conditioned on (i) and consistent with the dialogue flow. Evaluating state-of-the-art DST models on MultiWOZ dataset with CoCo-generated counterfactuals results in a significant performance drop of up to 30.8% (from 49,4% to 18.6% in absolute joint goal accuracy. In compari- son, widely used techniques like paraphrasing only affect the accuracy by most 2%. Human evaluations show that COCO-generatee conversations perfectly re- flect the underlying user goal with more than 95% accuracy and are as human-like the original conversations, further strengthening its reliability and promise be adopted as part the robustness evaluation of DST models.
INTRODUCTION
Task-oriented dialogue (TOD) systems have recently attracted growing attention and achieved sub- stantial progress (Zhang et al., 2019b] Neelakantan et al. 2019 Peng et al. 2020 Wang et al.) 2020b a). partly made possible by the construction of large-scale datasets (Budzianowski et al. 2018|Byrne et al., 2019 Rastogi et al. 2019). Dialogue state tracking (DST) is a backbone of TOD systems, where it responsible for extracting the user's goal represented (c.g., (area, center), (food, British)), as illustrated in the upper part of Figure []] The DST module's output is treated the summary of the user's goal far in the dialogue and directly consumed by the subsequent dialogue policy component to determine the system's 'ss next action and response. Hence, the accuracy of the DST module is critical to prevent downstream error propagation (Liu and Lane 2018), affecting the end-to-end performance of the whole system.
set of slot-value pairs
With the advent of representation learning in NLP (Pennington al. Radford et al.) 2019), the accuracy of DST models has increased from 15.8% (in 2018) to 55.7%
2014 Devlin al. 2019
(in 2020). While measuring the held-out accuracy is often useful, practitioners consistently overes- timate their model's generalization (Ribeiro et al. 20200) Patel et al. 2008) since test data is usually collected in the same way as training data. In line with this hypothesis, Table [ demonstrates that there is substantial overlap of the slot values between training and evaluation sets of the MultiWOZ DST benchmark (Budzianowski et al., 2018). In Table 2 we observe that the slot co-occurrence distributions for evaluation sets tightly align with that of train split, hinting towards the potential
"Equal Contribution. Work was done during Shiyang's 's intersship at Salesforce Research. Code available at attttps: / wtw..ccoo/ccc.c.ccccccccccc...ct wawaaaaaafaoo www.aaaaaaeee/ooooooooooo ctt.//ooooo333/ dst

---

Published as a conference paper at ICLR 2021
data
sttraction-name
hotel-name
restaurant-name
taxi-departure
taxi-destination
train-doparture
train-dcstinatioo
dev
94.5
96.4
97.3
98.6
98.2
99.6
99.6
lest
96.2
98.4
96.8
95.6
99.5
99.4
99.4
Table 1: The percentage (%) of domain-slot values in dev/test sets covered by training data,
slot name
data
area
book day
book time
Food
name
price range
train
1.9
38.8
39.2
2.1
16.4
1.5
book people
dev
1.9
38.9
38,9
1,9
16.3
2.2
lest
2.7
36.9
37.7
1.6
18.7
2.4
Table 2:
Co-occurrence distribution(S)) (%) of book people slot with other slots in restaurant domain within the same user utterance. It rarely co-occurs with particulars slots food), which hinders the evaluation of DST models on realistic user utterances such as "I want to book Chinese restaurant for 8 people."
limitation of the held-out accuracy in reflecting the actual generalization capability of DST models. Inspired by this phenomenon, we aim to address and provide insights into the following question: how well do state-of-the the-art DST models generalize to the novel but realistic scenarios that are not captured well enough by the held-out evaluation set?
Most prior work (lyyer et al., 2018; Jin et al. 2019) focus on adversarial example generation for robustness evaluation. They often rely perturbations made directly on test examples in the held- out set and assume direct access to evaluated models' gradients or outputs. Adversarial examples generated by these methods are often unnatural or obtained to hurt target models deliberately. It is imperative to emphasize here that both our primary goal and approach significantly differ from the previous line of work: (i) Our goal is to evaluate DST models beyond held-out accuracy, (ii) We leverage turn-level structured meaning representation (belief state) along with its dialogue history as conditions to generate user response without relying on the original user utterance, (iii) Our approach is entirely model-agnostic, assuming no access to evaluated DST models, (iv) Perhaps most importantly, aim to produce novel but realistic and meaningful conversation scenarios rather than intentionally adversarial ones.
We propose controllable counterfactuals (CoCo) as principled, model-agnostic approach gener- ate novel scenarios beyond the held-out conversations. Our approach is inspired by the combination of two natural questions: how would DST systems react to (I) unseen slot values and (2) rare but re- alistic slot combinations? CoCo first encapsulates these two aspects under unified concept called counterfactual goal obtained by a stochastic policy of dropping and adding slots to the original turn- level belief state followed by replacing slot values. In the second step, CoCo conditions on the dialogue history and the counterfactual goal to generate counterfactual conversation. We cast the actual utterance generation as conditional language modeling objective. This formulation allows us to plug-in a pretrained encoder-decoder architecture (Raffel et al.] 2020) as the backbone that powers the counterfactual conversation generation. We also propose strategy to filter utterances that fail to reflect the counterfactual goal exactly. We consider value substitution (VS), as presented in Figure [ special CoCo case that only replaces the slot values in the original utterance with- out adding or dropping slots. When we use VS as a fall-back strategy for CoCo (i.e., apply VS when CoCo fails to generate valid user responses after filtering), we call it CoCo+.
Evaluating three strong DST models (Wu et al. 2019 Heck et al. 2020 Hosseini-Asl et al. 2020) with our proposed controllable counterfactuals generated by CoCO and CoCo+ shows that the performance of each significantly drops (up to 30.8%) compared to their joint goal accuracy on the original MultiWOZ held-out evaluation set. On the other hand, we find that these models are, in fact, quite robust to paraphrasing with buck-translation. where their performance only drops up to 2%. Analyzing the effect of data augmentation with CoCo+ shows that it consistently improves the robustness of the investigated DST models on counterfactual conversations generated by each of VS, CoCo and CoCo+. More interestingly, the same data augmentation strategy improves the joint goal accuracy of the best of these strong DST models by 1.3% on the original MultiWOZ evaluation set. Human evaluations show that CoCo-generatte the underlying user goal with more than 95% accuracy and are found to be quite close to original conversations in terms of their human- like scoring. This further proves our proposed approach's 's reliability and potential to be adopted as part of DST models' robustness evaluation.
counterfactual conversations perfectly reflect
2

---

Published as a conference paper at ICLR 2021
Dialogue Flow
Turn-level Belief State
Dialogue-level Delief State
[8ystmm] Hallo, how can help you?
need find restaurant
Creetaurent, center>
<reabeurant, center'>
[Beerj It serve
Bystem]: have many optional you any preference?
crestburant, food, Britishs <rasburani,, book time, 18:00>
Kreadsurant, food, Britash,
Sreataurant, book time, 18:00
[Systemj: have many optionsl Do you preference?
crestburant, food, Chinesen creataurant, book time, 17:00>
crestaurant, area, centert, creedxurant, food, Chinase
Creabsurant book time. 17:40-
2
(CoCa): It hould serve Chinsss food and Ined to book tuble for people.
[System]:
have many options! Do you have any proference?
Kreetaurent, food, Chinassi, Krastaurent, book people, 2>
Sheadaurant, anea, center", cresdsurant, food, Chtiness spesdaurant, book people,
Figure 1:
The upper left is dialogue example between user and system with its turn-level and dialogue-level belief states the upper right. The lower left valid user utterance variations generated by VS and CoCo with their corresponding belief states derived from the original ones on the right.
2 RELATED WORK
Dialogue State Tracking. DST has been core component in current state-of-the-an TOD systems. Traditional approaches usually rely on hand-crafted features or Nomain--seciife lexicon (Henderson jet al. 2014; Wen et al., 2017) and require a predefined ontology, making them hard to extend to unseen values. To tackle this issue, various methods have been proposed. Gao et al. (2019) treats DST as a reading comprehension problem and predicts slot values with start and end positions in the dialogue context. Zhang et al. (2019a) proposes DS-DST, dual-strategy model that predicts values in domains with a few possible candidates from classifiers and others from span extractors. Furthermore, HHeck et al. 2020) proposes TripPy, a triple copy strategy model, which allows it to copy values from the context, previous turns' predictions and system informs.
An alternative to classification and span prediction is value generation. [Wu et al. (2019) generates slot values with a pointer generator network (See et al. (2017) without relying on fixed vocabularies and spans. (Hosseini-Asl al. 2020) models DST conditional generation problem and directly finetunes GPT2 RRadford et al.
2019) on DST task and achieves state-of-the-arg on the MultWOZ..
Adversarial Example Generation. Adversarial example generation has been commonly studied in computer vision (Szegedy et al. 2014 Goodfellow et al. 2015). Recently, it has received growing attention in NLP domain as well. Papernot el al. (2016) finds adversarial examples in the embedding space, and then remapped them to the discrete space. Alzantot et al. (2018) proposes a population- based word replacing method and aims to generate fluent adversarial sentences. These methods often edit the original data greedily assuming access to the model's gradients or outputs besides querying the underlying model many times (Jin et al.] 2019). Alternative line of work investigates generating adversarial examples in model-agnostic way. Lyyer et al. (2018) proposes generate adversarial paraphrases of original data with different syntactic structures. Jia and Liang (2017) automatically generates sentences with key word overlappings of questions in SQuAD (Rajpurkar et al. 2016 to distract computer systems without changing the correct answer or misleading humans.
Although different methods have been proposed to evaluate the robustness of NLP models, majority of the prior work in this line focus either on text classification, neural machine translation or reading comprehension problems. Perhaps the most similar existing work with ours are (Einolghozati al., 2019) and (Cheng et al.] (201)). EEnolghozaa et al.] (2019) focuses on intent classification and slot lagging in TOD while Cheng et al. (2019) targets at synthetic competitive negotiation dialogues (Lewis et al. 2017) without DST component. In this work, however, we focus on evaluating a core component of state-of- -fate-of-thh-arr TOD, DST, on the widely used benchmark, MultiWOZ. To the best of our knowledge, ours is the first work to systematically evaluate the robustness of DST models.
3 BACKGROUND
Multi-domain DST task definition. Let X, = (U U ... (U?7 UT denote a sequence of turns of a dialogue until the t-th turn, where U and U (1 i t) denote system and user utterance the i-th turn, respectively. In multi-domain DST. each turn (U .U!") talks about specific domain (c.g.. hotel). and a certain number of slots (e.g., price range) in that domain. We denote all N possible domain-slot pairs as S (S1, The task is to track the value for each
3

---

Published as a conference paper at ICLR 2021
S ( N) over X, (e.g., hotel-price range, cheap). Belief states can be considered two granularities: turn-level (Ly) and dialog-level (Bt). L, tracks the information introduced in the last turn while B, tracks the accumulated state from the first turn to the last. As illustrated in the upper part of Figure | when the dialogue flow arrives at the second turn, B2 becomes {rrestaurant-arel center), (restaurant-fodd, British), (restaurant-book time, 18:00)), while T.2 is { {restaurant-food British), (restaurant-book time, 18:00), essentially tracking the update to B, by the last turn.
Problem definition. Given a tuple < Xt, Lt B, >, our goal is to generate a new user utterance Ü to form novel conversation scenario X; {(U} U .... (U²", ((rr)) by replacing the original user utterance U with U To preserve the coherence of dialogue flow, we cast the problem as generating an alternative user utterance Ün conditioned on modified L, derived from original turnllevll belief state L; in way that is consistent with the global belief state B, This formulation naturally allows for producing a new tuple < Xx, L,, B, > controllable by Li, where B, is induced by B, based on the difference between L2 and L. As illustrated in the lower part of Figure 1 U is replaced with the two alternative utterances that are natural and coherent with the dialogue history. We propose to use the resulting set of < X,, L,, B₁ > to probe the DST models.
Paraphrase baseline with back-translation. Paraphrasing the original utterance U. is natural way to generate Üneer With the availability of advanced neural machine translation (NMT) mod- els, round-trip translation between two languages (i.c., back- translation (BT)) has become a widely used method to obtain paraphrases for downstream applications (Yu et al. 2018). We use publicly available pretrained Bnglish~>Germen
(log(gle) (g)) and (terman-FEnglh We translate U!TE from English to German with a beam size K, and then translate each of the A hypotheses back to English with the beam size K. Consequently, we generate Kd paraphrase candi- dates of ÜसSA and then rank them according to their round- trip confidence score log(g) e) + log(elg). As paraphrases are expected to preserve the meaning of Uuer we set Le L, and B, Be.
(log(elg) NMT models
CoCo
As illustrated in Figure 2. CoCo consists of three main pillars. We first train a conditional user utterance generation model po(U,"" I4) using original dialogues. Secondly, modify L, into a possibly arbitrary L, by our counterfactual goal generator. Given L, and U we sample UT ~ pa (U L.) with beam search followed by two orthogonal filtering mechanisms to further eliminate user utterances that fail to reflect the counterfactual goal i.-
4.1 VALUE SUBSTITUTION
A robust DST model should correctly reflect value changes in user utterances when tracking user's goal. However, slot-value combinations, e.g. (restaurant-book time, 18:00), in evaluation sets limited and even have significant overlaps with training data as shown in Table To evaluate DST models with more diverse patterns, we propose a Value Substitution (VS) method to generate Usar. Specifically, for each value of Sy in Lt, if the value only appears in U rather than U we allow it to be substituted. Otherwise, we keep it as is. This heuristic is based on the following three observations: (1) if the value comes from U e.g. TOD system's recommendation of restaurant food, changing it may make the dialogue flow less natural and coherent (2) if it never appears in the dialogue flow, e.g. yes of hotel-parking changing may cause belief state label errors (3) if it only appears in UNie it is expected that changing the value won't 't cause issues in (1) and (2).
For values that can be substituted, new values are sampled from a Slot- Value Dictionary, predefined value set for each domain-slot. These new values are then used to update their counterparts in U L, and B,. We defer the details of value dictionary to section 4.2 After the update, we get t!!er,
L, and B,, and can use < Xe, L,, B, > to evaluate the performance DST models. An example of how VS works is illustrated in the lower part of Figure I At the second turn, as British and 18:00 are in Lg and only appear in U?yar rather than U we can replace them with Chinese and 17:00 that
4

---

Published as a conference paper at ICLR 2021
Utterance Generation Model During Training Phase
During
Bys] have many optionsl Do you have any preference?
onkenrdiont Chaas, Inder. Noaanedisce peotic 1.2...
have many optional have www.eeyppooeeeeoooo preference) Sandeneenn
Counterfactull Doal Generator
[Bete]] food, Britssh cisetaurant book 18:30>
creatuurant book people,
resta matauraass-ooo food ruttaret cock
Search
[Uner] needs to narv British food
lhe
18:00.
1. want book table at Chisoss risturran.
[Classifier Filter d Volie Match Fitter
people
Brouut system: hove many optional you have any proferencc? state: [Outpulj:: neadie British food like reservation for 18:00
[Cleseiier Filter Slat- Matcch Filter:
3. Yes, loent to book table for 2 Chipsnn restaurant. [Cossiffor ai Volue
The overall pipeline of CoCo. The very left part represents the training phase of utterance generation model, where the concatenation of U22 and L, processed by the encoder, which the deceder then conditions on to generate the user utterance Ut The input and output of this model is shown within the box at the lower-left. The right part depicts the inference phase, where the counterfactual goal generator first modifies the original belief L, fed from the left part into new one L., which is then fed to the trained utterance generator along with the same conversation history to generate Üi by beam search followed by filtering undesired utterances. Note that conversational turns in inference phase don't 't have to originate from training phase.
Figure 2:
are sampled from a slot-value dictionary, respectively, to get Ün L2 and X, without interrupting the naturalness of the dialogue flow.
4.2 CONTROLLABLE COUNTERFACTUAL GENERATION
Back-translation (BT) and value-substitution (VS) provide controllability at different granularities. BT only provides syntactic variety while preserving the meaning, hence the belief state. VS can only replace the values of the existing slots in an utterance while still having to exactly retain all the slots. However, neither of them are able to explore conversations with even slightly modified set of slots. We propose a principled approach to unlock the capability of conversation generation that generalizes beyond just transformation of existing utterances. We cast it as task of generating novel user utterances (UPar) from a given conversation history (U,))) and a turn-level user goal (T,))-
We propose tackle this problem with conditional generation model that utilizes pre- trained encoder-decoder architecture (Raffel et al. [020;; Lewis et al.] (2020] to approximate p(U,""11 Lt). where the concatenation of U. and Le is used input the encoder and U, use is set to be the target sequence to be generated by the decoder, as illustrated in the lower-left of Figure 2. To learn this distribution, we factorize it by chain rule (Bengio et al.. (003) and train neural network with parameters 0 to minimize the aggregated negative log-likelihood J gen over each dialogue turn tuple (U.Yh. Lt, U where UT (URHH Uuar
......., UTSr (,n,) and Uher is its k-th token:
12g
Po Po(U U L.)
II Po UPS ,L), Jgen
Σ log po (U U, U L.) (1)
Once the parameters 9 of the goal-conditioned utterance generation model pe are learned from these tuples, gives us the unique ability to generate novel conversation turns by plugging in an arbitrary but consistent counterfactual goal f., derived from T., An example of how the counterfactual goal generator operates is shown in the middle part of Figure 2 The counterfactual goal generator has three components, namely operation, slot-value dictionary and clot-comminttion dictionary.
Operation decides to apply which combination of the following three meta-operations, namely drop, change and add on L, Drop is used to remove values from a non-empty slot in L,- Change borrows the same operation in VS, substitute existing values. Add allows to add new domain- slot values into Lt,, giving us the power of generating valid but more complicated irus..
Mooe details can be found in Appendix B.1
5

---

Published as a conference paper at ICLR 2021
Slot- Value Dictionary has pre-defined value set S3at for each Sy. Once change and/or add meta- operation activated for Sy. counterfactual goal generator will randomly sample a value from S
Nlot-Combination Dictionary has a predefined domain-slot set Sydd for each Sj. When add meta- operation is activated, counterfactual goal generator will sample domain-slot from the intersection among all Stddd, where S, has non-empty values within Lt Once a new domains-slot is sampled, its value will then be sampled from its corresponding value set as defined in slot-value dictionary.
Given L., the counterfactual goal generator first takes Le as its input, and sequentially applies drop, change and add to output L, Given L; and U."e we can sample U ~ pe (Ü | L() with beam search. We use a rule-based method to get B, of X.. Specifically, we obtain - by calculating the set difference of B, and L, Given B and L., we update the domain-slot in _ if its value in L, is not none, otherwise we keep its value as is in B following (Chao and Lane 2019 After the update, we get B, and use the dialogue-level label of Xt.
4.3 FILTERING
We have presented methods to generate Úpn, but how do make that the generated utterance correctly reflects the user goal represented by f., To motivate our methods, we take an example generated by beam search located at the lower right of Figure for for illustration. In this example, the first hypothesis doesn't 't include value 2 for restaurant -boo people that is within L. On the contrary, the second hypothesis includes value 18:00 for restaurant-bodd time that is not part of f., We call these two phenomenons de- de-generation and over- pver-generation, respectively. Filtering candidates with these issues thus an important step to make sure (Ut', Ú perfectly expresses the user goals in L., We propose two filtering methods, namely slot-value match filter and classifier to alleviate de-generation and over-generation issues, respectively.
Slot-Valus Match Filter. To tackle with de-generation issue, choose a subset of values in i. (values that should only appear in Ü rather than U²"") to eliminate candidates that fail to contain all the values in the subset 4 In Figure 2 the first hypothesis from the beam search output will be eliminated by this filter because it does not include the value 2 for restaurant- bookk people in L.
Classifier Filter. As shown in Table 2 the slot restaurant- book people frequently appears together with restaurant-book time the data used to train our generation model pe (U?" L.), which may cause the resulting generation model to fall into over-generation issue. To deal with this over- generation problem, we propose to use N-way multi-label classifier to eliminate such candidates. We employ BERT-base (Devlin et al., 2019) its backbone:
H BERT([CLS] @ [X,-1] @ [SEP] @ [ @ (17HT] E Rum
(2)
where Hois E dade is the representations of CLS token of BERT with dimension demb- We then feed Hçis into linear projection layer followed by S Sigmoid function:
N
P= Sigmoid E RRN, Jas = N
Σ (Y, logp, + YY) log(1 - PJ)) (3)
j=1
where W E N is the trainable weight of the linear projection layer and Py is probability that slot Sj appears at 1-th turn of X, with Y, as its label. The classifier is trained with Jak. i.e. the mean binary cross entropy loss every slot S, and achieves a precision of 92.3% and a recall of 93.5% on the development set During inference, the classifier takes X, as input and predicts whether a slot S; appears at t-th turn or not with threshold 0.5. We use this filter to eliminate generated candidates for which the classifier predicts at least one slot S mentioned in (U ((*er) while Sj & L. In Figure our classifier filter eliminates the second hypothesis from the output of beam search because L, does not contain the slot eestaurant-ood time while it is mentioned in the generated utterance.
Forr hotel-parking and hotel-internet, we use parking and wifi as their corresponding values for filtering. We defer further details of the classifier Appendix F.2

---

Published as a conference paper at ICLR 2021
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
We consider three strong multi-domain DST models to evaluate the effect of CoCo-generated coun- terfactual conversations in several scenarios. TRADE (Wu et al.] 2019) builds upon pointer gener- ator network and contains slot classification gate and state generator module generate states. TRIPPY (Heck et al. 2020) introduces a classification gate and a triple copy module, allowing the model copy values either from the conversation context or previous turns' predictions or system informs. SIMPLETOD (Hosseini-As] et al., (2220) models DST as conditional generation problem with conversation history as its condition and belief state as its target and finetunes on GPT2.
Evaluation. We train each of these three models following their publicly released implementations on the standard train/dev/test split of MultiWOZ 2.1 (Eric et al.) 2019). We use the joint goal accuracy to evaluate the performance of DST models. It is 1.0 if and only if the set of (domain-slot, value) pairs in the model output exactly matches the oracle one, otherwise 0.
Slot- Value Dictionary. We carefully design two sets of slot-value dictionaries to capture the effect of unseen slot values from two perspectives, namely in-domain (/) and out-of-domain (0). / is a dic- tionary that maps each slot to set of values that appear in MultiWOZ test set, but not in the training set 6 fall completely outside of the MultiWOZ data for the slots (e.g., hotel-name, restaurant- etc.). Otherwise, we follow a similar fall-back strategy for slots (e.g.. hotel-internet) with no possible external values beyond the ones (e.g., yes and no) in the original data.
On the other hand, we construct O using external values (c.g.. hotel names from Wikipedia) that
Cott-oomnnn Dictionary. As illustrated in Table 2 held-out evaluation set follows almost the same slot co-occurrence distribution with training data. This makes it difficult to estimate how well DST models would generalize on the valid conversation scenarios that just do not obey the same distribution. CoCo's flexibility at generating a conversation for an arbitrary turn-level belief state naturally allows us to seek an answer to this question. To this end, we design three slot combina- tion dictionaries, namely freq, neu and rare. slot combination dictionary directly controls how different slots can be combined while generating counterfactual goals. As suggested by their names, freq contains frequently co-occurring slot combinations (e.g., book people is combined only with book day and book time slots), while rure is the opposite of freq grouping rarely co-occurring slots together, and neu is more neutral allowing any meaningful combination within the same domain
5.2
MAIN RESULTS
Before reporting our results, it is important to note that several different post-processing strategies are used by different DST models. To make fair comparison across different models, we follow the same post-processing strategy employed by SIMPLETOD evaluation script for TRADE and TRIPPY as well. We summarize our main results in Figure 3 While all three DST models are quite robust to back-translation (BT) their performance significantly drop counterfactual conversations gener- ated by each of VS. CoCo and CoCo+ compared to MultWWZZ held-out set accuracy (original).
Unseen Value Generalization. We analyze the effect of unseen slot values for the two dictio- naries (I and O) introduced in the previous section compared to the original set of slot values that have large overlap with the training data. Results presented on the left part of Figure 3 that the performance of DST models significantly drops up to 11 11.8% compared to original accuracy even on the simple counterfactuals generated by VS strategy using in-domain unseen vlou-value dictionary (I). Furthermore, using out-of-domain slot-value dictionary (O) results in about 10% additional drop in accuracy consistently across the three models. Consistent and similar drop in accuracy suggests that TRADE, SIMPLETOD, and TRIPPY are almost equally susceptible to unseen slot values.
Generalization to Novel Scenarios. The right section of Figure 3 presents the main results in our effort to answer the central question we posed at the beginning of this paper. Based these re-
When this set is empty for a slot (c.g., hotel-area), we use the set of all possible values (c.g., center, east, west, south, north) for this slot from training data. Please see Appendix ffor for further details.
Please see Appendix H for further details.
SSiilarr to CoCo, we back-translate only the turns with non-empty turn-level helief states and apply slot- value match filter. We fall back original user utterance if none of the paraphrases passes the filter.
7

---

Published as a conference paper at ICLR 2021
$4.0
56.1
$2.8
TRADE SimpleTod TripPy
50
43.4
40.0
37.
39.1
34A
30
22.4
21.6
18.
28.5
Originl
CoCoofro))
CoConttog)
CoCofneu)
GoCor(ned)
CoCafrend
CoCerfrend)
Figure 3: Joint goal accuracy (%) across different methods. "Original" refers to the results the original held-out test set. * denotes results obtained from in-domain unseen slot-valuc dictionary (/). VS, CoCo and CoCo+ results use out-of-domain slot-value dictionary (O). For brevity, we omit CoCo
CoCo+ results using in-domain slot-value dictionary. See Appendix C for the full results. freg, neu, and rare indicate which slot-combination dictionary is used. Lower bound refers to the percentage of correct predictions turns with empty turn-level belief state over original held-out
sults, we see that state-of- of-the-art DST models are having serious difficulty generalizing to novel scenarios generated by both CoCo and CoCo+ using three different slot combination strategies. The generalization difficulty becomes even more serious on counterfactuals generated by CoCo+. expected, the performance drop consistently increases as we start combining less and less fre- quently co-occurring slots (ranging from freq to rare) while generating our counterfactual goals. In particular, CoCo+(rare) counterfactuals drops the accuracy of TRADE from 49.4% to 18.6%, pushing its performance very close to its lower bound of 13.8%. Even the performance of the most robust model (TRIPPY) among the three drops by up to 25.8%, concluding that held-out accuracy for Mfate-offie oatie--f-hh--ann the-art DST models may not sufficiently reflect their generalization capabilities.
Transferability Across Models. As highlighted before, a significant difference and advantage of our proposed approach lies in its model-agnostic nature, making it immediately applicable for eval- uation of any DST model. As can be inferred from Figure 3 the effect of CoCO-generated coun- terfactuals the joint goal accuracy is quite consistent across all three DST models. This result empirically proves the transferability of CoCo, strengthening its reliability and applicability to be generally employed as a robustness evaluation of DST models by the future research.
5.3 HUMAN EVALUATION
Human likeliness
We next examine the quality of our generated data from two perspectives: "human likeliness" and "turn-level belief state correctness".. The human likeliness evaluates whether user utterance is fluent and consistent with its dialog context. The tum-level belief state correctness evaluates whether (U?7, Ú!"") exactly expresses goals in L. Both metries are based on binary evaluation. We randomly sample 100 turns in the orig- inal test data and their corresponding CoCo-generated ones. For the CoCo-generater data, we have two different settings to examine its quality. The first is use the original turn-level belief state to generate user utterance, denoted by CoCo(ori). The second setting is to verify the quality of the conversations generated by CoCo(freq)- CoCo(neu)- and CoCo (rare) as they hurt the DST models' accuracy significantly as shown in Figure For each result row reported in Table 3 ask three individuals with proficient English and advanced NLP background to conduct the evaluation, and use majority voting to determine the final scores.
Correctness
Human CoCo(er))
87%
85%
90%
91%
CoCotfreg)
90%
99%
CoCorneu)
79%
98%
CoCotrare)
82%
96%
Table 3: Human evaluation.
We can see that CoCo(ori) generated conversations are almost human-like as original conversa- tions. Furthermore, CoCo(ori) generated slightly more "correct" responses than the original utter- ances in MultiWoZ 2.1. A presumable reason is that annotation errors exist in MultiWoZ 2.1, while our CoCo are trained on recently released cleaner MultiWoZ 2.2, making generated data have higher quality. In addition, all three variants of the CoCo-generated conversations consistently outper-
8

---

Published as a conference paper at ICLR 2021
TRADE Simppeeed Trppy
56.3
$1.3
33.4
37.9
36.0
31.8
26.3
300
27.3
22.3
26-2
Origial
Coto-Jive)
Figure 4: Comparison of retrained DST models (indicated by ) CoCo+(rarc- training data with their counterparts trained original MultiWOZ train split.
form human response in terms of the turn-level belief state correctness. Although CoCo(neu) and CoCe(rare) are slightly less human-like than the original human response, (oCooffra)-generatee utterances have similar likenn-s original ones. These results demonstrate the effectiveness of our proposed approach in generating not only high-fidelity but also human-like user utterances, proving its potential be adopted part of robustness evaluation of DST models.
5.4 ANALYSIS OF CoCo+ AS DATA AUGMENTATION DEFENSE
So far, we have focused on the generalization capability of DST models on CoCo -eCo--geeetted conver- sations using different slot-value and slot-combination dictionaries. We have observed that all three DST models are consistently most susceptible conversations generated by CoCo+(rare) strategy. Instead, we now seek to answer the following question: Would using conversations generated by CoCo+(rare) to augment the training data help these DST models in better generalizing to unseen slot values and/or novel scenarios? Towards exploring this direction slot value dictionary (train-O) similar out-of-domain unseen slot-value dictionary (O). For a fair comparison, we make sure that the slot values in train-O (please refer to Appendix [ for the complete dictionary) do not overlap with the one (O) used for generating test conversations.
principled way, we design
We first retrain each DST model on the MultiWOZ training split augmented with CoCo+(rare)- generated conversations using train-O slot-value dictionary. Retrained DST models then evalu- ated on original test set as well as on the counterfactual test sets generated by VS and various ver- sions of CoCo+. Results presented in Figure show that retraining on the CoCo=(rar)-augmentt training data improves the robustness of all three DST models across the board. Most notably, it rebounds the performance of TRIPPY on CoCo+(rare)ggeerraa significantly closing the gap with its performance (61.3%) on the original test set. We also observe that retrained DST models obtain an improved joint goal accuracy on the original MultiWOZ test set compared to their counterparts trained only on the original MultiWOZ train split, further validat- ing the quality of -ener-geee conversations. Finally, we would like to highlight that retrained TRIPPY achieves 62.6% joint goal accuracy, improving the previous state-of-the-art by 1.3%. We leave the exploration of how to fully harness CoCo as data augmentation approach as future work.
test set from 35.5% to 56.2%,
CONCLUSION
We propose principled, model-agnostic approach (CoCo) to evaluate dialogue state trackers be- yond the held-out evaluation set. We show that state-of-the-arr DST models' performance signif- icantly drop when evaluated the CoCO-generated conversations. Human evaluations validate that they have high-fidelity and are human-like. Hence, we conclude that these strong DST models have difficulty in generalizing to novel scenarios with unseen slot values and rare slot combina- tions, confirming the limitations of relying only on the held-out accuracy. When explored as data augmentation method, CoCo consistently improves ofate-of-the-an DST models not only the CoCo-genan -ece-geeeatted evaluation set but also on the original test set. This further proves the benefit and potential of our approach be adopted part of more comprehensive evaluation of DST models.

---

Published as a conference paper at ICLR 2021
REFERENCES
M. Alzantot, Y. Sharma, A. Elgohary, B.-J. Ho, M. Srivastava, and K.-W Chang. Generating natural language adversarial examples. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890-2896, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v//D18- 1316. URL https: www aclweb.
hog/aathhloogy/D155
Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. A neural probabilistic language model. J. Mach. Learn. Res., 3(mull):1377-1155: Mar. 2003. ISSN 1532-4435.
P. Budzianowski, T.-H. Wen, B.-H. Tseng, I. Casanueva, S. Ultes, O. Ramadan, and M. Gašić. Mul- tiWOZ large-scale multi-domain wizard-of-Or dataset for task-oriented dialogue modelling- In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
B. Byme, K. Krishnamoorthi, C. Sankar, A. Neelakantan, B. Goodrich, D. Duckworth, S. Yavuz, A. Dubey, K.-Y. Kim, and A. Cedilnik. Taskmaster-l: Toward a realistic and diverse dialog dataset. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), 2019.
G. Chao and I. Lane. Bert-dst: Scalable end-to-end dialogue state tracking with bidirectional encoder representations from transformer. In INTERSPEECH. 2019.
L. Chen, B. Lv, C. Wang, S. Zhu, B. Tan, and K. Yu. Schema-guided multi-domain dialogue state tracking with graph attention neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7521-7528 Apr. 2020. doi: 10.1609/aaai, v34005.645555 URL https://
ojs. aaai. org index. php/AAAI AAAI wttp:aattttceeeeee/eeeeee 6250
M. Cheng, W. Wei, and C.-J. Hsich. Evaluating and enhancing the robustness dialogue sys- tems: A case study
negotiation agent. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3325-3335, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.19653/ URL https: // www. ac www.wllwwbbb..... org/ wwt.ooathooooyyooo Nooooolyyy/N1/333
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre Pre-training of deep bidirectional trans- formers for language understanding. Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
A. Einolghozati, S. Gupta, M. Mohit, and R. Shah. Improving robustness of task oriented dialog systems. ArXiv, abs/191 1.05153, 2019.
M. Eric, R. Goel, Paul, A. Kumar, A. Sethi. A. K. Goyal, P. Ku, S. Agarwal, Gao, and D. Z. Hakkani-Tür. Multiwoz. 2.1: Multi-domain dialogue state corrections and state tracking baselines. ArXiv, abs/1907.01 669,, 2019.
S. Gao, A. Sethi, S. Agarwal, T. Chung, and D. Z. Hakkani-Tüi: Dialog state tracking: A neural reading comprehension approach. ArXiv, abs/1908.00 01940,9996, 2019.
I. Goodfellow, Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http:://axiv org/
abs/1412.6577 6572
M. Heck, C. van Niekerk, N. Lubis, C. Geishauser, H.-C. Lin, M. Moresi, and M. Gasic. TripPy:
A triple copy strategy for value independent neural dialog state tracking. In Proceedings of the 2 11th Annual Meeting of the Special Interest Group Discourse and Dialogue, pages 35 44, 1st virtual meeting, July 2020. Association for Computational Linguistics. URL https: :tto:://www wwww.
acl weeb.org/antho logy /200 1tt.0000..iiiaal100aa.1.11
aiiiilll1.4
10

---

Published as a conference paper at ICLR 2021
M. Henderson, B. Thomson, and S. Young. Word-bases dialog state tracking with recurrent neural networks. In Proceedings of the 15th Anmual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 292-299, Philadelphia, PA, U.S.A., June 2014. Association for Computational L Linguistics. doi: 10.31 (0.3115/11W1444544 URL https: http//wwacccomm web. org/
anthol.ogy/81444344
E. Hosseini-Asl, B. McCann, C.-S. Wu, S. Yavuz, and R. Socher. A simple language model for task-oriented dialogue, 2020.
M. Iyyer, J. Wieting, K. Gimpel, and L. Zettlemoyer. Adversarial example generation with syn- tactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long Papers). pages 1875-1885, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 2418665/11/N1-- org/ antha NoarthhoooyyN188-11
URL BtLhhttp:: www aclweb.
R. Jia and P. Liang. Adversarial examples for evaluating reading comprehension systems. In Pro- ceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021 1031, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguisiis.. doi: 10.18653/v1/DI 1/177-121.. URL https:: www. aclweb. org/
http://wwww....///..toooooo////. arthology/DD7-1121
D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits. Is bert really robust? natural language attack on text classification and entailment. ArXiv, abs/1907. 2019.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
A. Kumar, P. Ku, A. K. Goyal, A. Metallinou, and D. Z. Hakkani-Tür, Ma-dst: Multi-attention based scalable dialog state tracking. ArXiv, abs/2002.08898, 2020.
H. T. Le, R. Socher, and S. Hoi. Non-autoregreesivee dialog state tracking. ArXiv, abs/2002.08024, 2020.
M. Lewis, D. Yarats, Y. Dauphin, D. Parikh, and D. Batra. Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference Empirical Methods in Nat- ural Language Processing, pages 2443-2453, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics. doi: 10. 1/D17-12//// org/ #xc/aantholooy/D17-2222
URL https: www. acl
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettle- moyer. Bart: Denoising sequence-to-ssequeece pre-training for natural language generation, trans- lation, and comprehension. ArXiv, abs/1910.13461, 2020.
J. Li, M. Galley, C. Brockett, J. Gao, and W. Dolan. A diversity-promootngg objective function for neural conversation models. ArXiv, abs/1510.03055, 2016.
Z. Lin, A. Madotto, G. I. Winata, and P. Fung. Mintl: Minimalist transfer learning for task-oriented dialogue systems. ArXiv, abs/2009. 12005, 2020.
B. Liu and I. Lane. End-to-end learning of task-oriented dialogs. In Annual Meeting of the Associ- ation for Computational Linguistics (ACL.), 2018
S. Mehri, M. Eric, and D. Hakkani-Tur. Dialoglue: A natural language understanding benchmark for task-oriented dialogue. ArXiv, abs/2009. 2020.
A. Neelakantan, S. Yavuz, S. Narang, V. Prasad, B. Goodrich, D. Duckworth, C. Sankar, and X. Yan. Neural assistant: Joint action prediction, response generation, and latent knowledge reasoning. In NewrlPS 2019 Converstional Al Workshop, 2019.
N. Papernot, P. McDaniel. A. Swami, and R. E. Harang. Crafting adversarial input sequences for recurrent neural networks. MILCOM 2016 2016 IEEE Military Communications Conference, pages 49-54, 2016.
11

---

Published as a conference paper at ICLR 2021
K. Patel, J. Fogarty, J. A. Landay, and B. Harrison. Investigating statistical machine learning tool for software development. In CHI, 2008.
B. Peng, C. Li, J. Li, S. Shayandeh, L. Liden, and J. Gao. Soloist: Few-shot task-oriented dialog with a single pre-trained regressive model, 2020.
Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, 2014. URL http: /ttt:://ww..r aclweb. org/ htwpg//nnhooogy... Lttplo/gy/111111111
A. Radford, J. Wu, R. Child, L. David, D. Amodei, and I. Sutskever. Language models are unsuper- vised multitask learners. OpenAl Blog, 2019.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100, 000+ questions for machine compre- hension of text. ArXiv, abs//6600005250 2016.
A. Rastogi, X. Zang. S. Sunkara, R. Gupta, and P. Khaitan. Towards scalable multi-domain conver- sational agents: The schema-guided dialogue dataset. arXiv preprint arXiv:I909) 05855, 2019.
M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguisties. Association for Computational Linguistics, 2020.
A. See, P. J. Liu, and C. D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 00.18653/v1/P17-110 org/ attaaaoooooooll-111111
URL https:: www. aclweb.
C. Szegedy, W. Zaremba, I. Sutskever, Bruna, Erhan, I. Goodfellow, and Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
URL
wwwp/////l/aaorrr..... wwwprrwwwi.ovv.ro.mee abs/ 111..111..... 6199..
J. Wang, Y. Zhang. T.-K. Kim, and Y. Gu. Modelling hierarchical structure between dialogue policy and natural language generator with option framework for task-oriented dialogue system. ArXiv, abs/2006.066 06000006614, 14, 2020a.
K. Wang, J.-F. Tian, R. Wang. X. Quan, and J. Yu. Multi- dialogue acts and response co- generation. In Annual Meeting of the Association for Computational Linguisties (ACL), 2020b.
T.-H. Wen, D. Vandyke, N. Mrkšić, M. Gašić, L. M. Kojas-Barahnne P.-H. Su, S. Ultes, and S. Young. A naseerk-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449, Valeniia, Spain, Apr. 2017. Association for Computa- tional Linguistics. URL httpt:tt///// //t::///wwwww. www. ac web. org/ ant ho 5
C.S. Wu, A. Madotto, E. Hosseini-Asl, C. Xiong, R. Socher, and P. Fung. Transferable multi- domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers). Association for Computational Linguistics, 2019.
A. W. Yu, D. Dohan, M. Luong, R. Zhao, K. Chen, M. Norouzi, and Q. Le. QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension. In 6th International Conference on Learning Representations (ICLR), 2018
X. Zang, A. Rastogi, S. Sunkara, R. Gupta, J. Zhang, and J. Chen. MultiWOZ 2.2 A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop Natural Language Processing for Conversational AI, pages 109-117, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18633/1122200mmp4omoww 1.13.
URL https: hts:://wwwww wwwww/wwwww.////./////www..// ac web. www.
12

---

Published as a conference paper at ICLR 2021
J. Zhang, K. Hashimoto, C.-S. Wu, Y. Wan, P. S. Yu, R. Socher, and C. Xiong. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. ArXiv, abs/1910.03544, 2019a.
Y. Zhang, Z. Ou, and Z. Yu. Task-oriented dialog systems that consider multiple appropriate re- sponses under the same context, 2019b.
13

---

Published as a conference paper at ICLR 2021
APPENDIX
A SLOT-LEVEL ANALYSIS
Closer Look at the Effect of CoCo+(rare) on TRIPPY. In Figure 5 take closer look at the robustness of TRIPPY through slot-level analysis across three major scenarios. Comparison of blue and orange lines reveals that counterfactuals generated by CoCo+(rare) consistently drops the performance of TRIPPY model (trained on the original MultiWOZ train split) across all the slots, significantly hurting the accuracy most slots in truin domain along with book day slot for hotel domain. On the other hand, comparing green and orange lines clearly demonstrates the effectiveness (rrc+(rarc) as a data augmentation defense (see Section 5.4 further details), assisting TRIPPY in recovering from most of the errors it made on CoCo+(rarc) evaluation set. In fact, it rebounds the joint goal accuracy of TRIPPY from 35.5% to 56.2% as presented more quantitatively in Figure
Ori- Ole-TripP--Ceenn
Ori-TriPy-CCCoo(rare)
Aug-TriPy-CCCoo-raree
texi-lesveAt
train-departure
destinestinntoon
train-ariueBy
train-day
taxi-ariveBy
.98
train destination
restaurant book people
train-lesvett
restaurant-bood
trainbookk people
restaurant boonuratt-book..oo time
attraction-area
restaurant food
attraction-name
restaurant pricerange
ettraction-type
restburant-name
hotel-type
hotellinternet
hotel book people
hotel-stars
hotel waok-bbook.eee day
hotel-pricerange
stay
hotel-parking
hotel-area
Figure 5: Slot-level accuracy analysis TRIPPY. "Ori- Tri-TripPy-Clean" (blue) and Cehi-TripPy-CoCo((mee (orange) denote TRIPPY (trained on original MuluWOZ training data) when evaluated against original test set and CoCo+(rare) generated test set, respectively. "Aug- "Aug-TripPy-Ce TriPayCCooo<rrre)" (green) indicates slot-level accuracy of TRIPPY after data augmentation (see Section Section5..4fo for further details) when evaluated against test set generated by CoCo+(rare).
14

---

Published as a conference paper at ICLR 2021
B ABLATION STUDY ON OPERATIONS
In 4, we present ablation results on three meta operations (i.e., drop, change, add) that are used to generate counterfactual goals. The result in the first corresponds the performance of three DST models on evaluation set generated by CoCo including all three meta operations along with the classifier filter. Each row analyzes the effects of the corresponding meta operation classifier by removing it from full models. From Table 4 we observe that removing drop operation from full models hurts the performance of the three models further. This may indicate that the investigated DST models are more vulnerable against user utterances including more slot combinations.
CoCo
TRADE
SIMPLETOD
TRIPPY
Full
26.2
31.6
42.3
-Drop
25.7
31.1
42.1
-Add
30.4
36.0
50.4
-Change
34.1
40.9
48.3
-Classifier
25.3
30.5
41.3
Table 4: Ablation study on the meta operations and classifier based filtering.
C FULL FIGURE FOR MAIN RESULT
15
Figure 6: Joint goal accuracy (%) across different methods. "Original" refers to the results on the original held-out test set. denotes results obtained from in-domain unseen slot-value dictionary (I) while other results use out-of-domain value dictionary (O).freq neu, and rare indicate which slot-combination dictionary is used.
D CoCo+ MULTI-ROUND DATA AUGMENTATION ON TRIPPY
Section 5.4 shows that CoCo+ as data augmentation (COCOAUG) improves TRIPPY'S joint goal accuracy by 1.3% when evaluated on the original test set following the post-processing strategy em- ployed by SIMPLETOD. In this section, we further extend previous single-round data augmentation into multiple rounds. Specifically, for each tuple < Xt, Le, B, in the original training set, we can generate multiple X,, Li, Be by sampling L, multiple times and utilizing CoCo+ generate corresponding X, and B,. With this approach, generated multiple X1, i,, B, combined with original < Xt, Lt, B, > can be used to train DST models.
We experiment with {1, 2, 4, 4, times data augmentation size original training data TRIPPY following its own default cleaning so that results with previous methods are comparable. Com- parison results with different baselines and data augmentation sizes are summarized in Table 5 When using more and more CoCo+ generated training data, TRIPPY gains benefits from more train- ing data and consistently improves over baselines. When using 8x CoCo+ generated training data, TRIPPY provides 5.49% improvement over its counterpart without data augmentation. Furthermore, achieves the new state-of-the-an join goal accuracy 2 outperforming CONvBERT-DG+MUn which uses open-domain dialogues and DialoGLUE (Mehri et al. 2020 as additional training data.
Code is available at
com/ sal http:///ooooooeeeeooooooooo///// ff
coco_aug __g
15

---

Published as a conference paper at ICLR 2021
Model
JOINT GOAL ACCURACY
DSTreader Gaa et al.
2019)
36.40%
TRADE Wu el al. 20
45.60%
MA-DST Kumar et al. 2020
51.04%
NA-DST Le et al. (0200)
49.04%
DST-picklist (Zhang
2019a)
53.30%
SST (Chen et al.
2020)
55.23%
(L.n MinTI.T1S-mmall
2020
50.95%
SimpleTOD (Hossemn Asi et al. 2020
55.76% 9
Conv (Mehri et al. 2020)
58.70% $4
TRIPPY (Heck et al. 2020)
55.04%*
+ COCOAUG (Ix)
56.00%
CoCoAUG (2x)
56.94%
+ CoCOAUG (4x)
59.73%
CoCOAUG (8x)
60.53%
Table 5: Joint goal accuracy results on MultiWOZ. 2.1 (Eric et al. 2019) of different methods. The upper part are results of various baselines and lower part are results of TRIPPY without or with 21,2, 2, 4, 8} times data augmentation size over original training data. t: results reported from (Zhang al. 2019a). $: results reported their original papers. *; results of our run based on their officially released code. 9: results need open-domaa dialogues and DialoGLUE data.
E MODEL DETAILS
E. THE DETAILS OF CONTROLLABLE GENERATION MODEL
We instantiate po (U U L,) with T5-smal1 Raffel al. 2020 and utilize MultiWOZ 2.2 as its training data since it's cleaner than previous versions (Zang et al. (020). During training, we use Adam optimizer (Kingma and Ba 2015) with initial learning rate 5e and set linear warmup to be 200 steps. The batch size is set to 36 and training epoch is set to be 10. The maximum sequence length of both encoder and decoder is be 100. We select the best checkpoint according lowest perplexity on development set.
E.2 THE DETAILS OF CLASSIFIER FILTER
the backbone of our classifier filter and train classifier filter with Adam optimizer (Kingma and Ba 2015) on MultiWOZ 2.2 since it's cleaner than previous ver- sions (Zang al. 2020). We select the best checkpoint based the highest recall development set during training process. The best checkpoint achieves a precision of 92.3% and recall of 93.5% on the development set of MultiWOZ 2.2 and, precision of 93.1 % and recall of 91.6% its original test set.
We employ BERT-basen CERT-bbas--ncaann
16

---

Published as a conference paper at ICLR 2021
F DIVERSITY EVALUATION
slot name
data
area
book day
book time
food
name
price range
entropy
Ori-test
36.9
37.7
18.7
0.57
book people
CoCo-test
3.6
38.5
25.2
15.6
14.8
2.2
0.65
Table 6: Original test set (Ori-test) and CoCo generated test set (CoCo-test) co-occurencce distribution( %6 comparisons of book people slot with other slots restaurant domain within the distribution entropy of CoCo-test higher than counterpart Ori-test with an upper bound 0.78 corr- sponding to uniform distrihution, meaning that CoCo-test more diverse compared Ori-test in terms slot combinations.
utterance. The
Data
Distinct-1 f Distinct-2
Distinet-3 1 Distinct-4 f
Ori-test
0.009
0.051
0.105
0.151
CoCo-test
0.009
0.053
0.113
0.166
Table Language diversity comparisons of data points between Ori-test and CoCo-test. We use unique n-gram ratio (Li et al. (016) our diversity metric. 1 represents higher number means more diversity. Overall, CoCo-test has similar (if not better) diversity scores compared to Ori-test.
17

---

Published as a conference paper at ICLR 2021
G GENERATED EXAMPLES BY CoCo
Dialogue Flow
Bystemmj:
CoCo-Turn
[CoCoo] need book bioht from now worh to san francisca people mondday.
<llght, destination franciscos, <llghtt people, <llight departure, new yorks, <llight, day, mondayo
Figure 7: Zero-shot generation ability of CoCo on flight domain, which is never seen during training.
Dialogue Flow
Turn-leval Bellief State
[Sysam]]
type
Tum
south unit g6 cambridge lolsure park, clifton road do you need
their phone number
[Ueer)
confirm address
postcade.
[Systam]:
cambridge
postoode cb17gy anythng else can for
[User]: wwant indian food center aren
creataurant, food, Indiank, Krestaurant, amss, centrex
[Bystemj:
g6: cambridge
Fallrr:
the postooda (7dy thair anything alse can you
people, food, book 18:30 Kreaburant, name, madarina palacao
day. monday>
[CoCo] yooy bool-- lable for people of on-mondayo
Re-eloe-
[System]
unit
Coco-Turn
[CoCo]: looking
called sheep
craetsurant, arao, northi, Kredfaurant, nama, sheap reabuurnnts, crestaurant, day, thursdayo
restaurant the north. need to book for thursday.
Figure 8: A success and failure example generated by CoCo with different slot-value combinations.
Dialogue
Tur-Jeevl Bellef State
[8ystem] tr814 will cost 9.80 pounds and arrivea 14:58 anything alsa you [User]:
stay..
Byystem what area of town would you like to be
[uer]]: dowant matter. just want
cheap gueathouse with wifl
<hotel, area, dontcare> <hotel, price range, cheap?, chotol, internet, yes?, <hotel, type, guesthouse>
[Bystemā: what area of town would you like to
CoCO.Tum
[C9tta """""" 166kthig 167" "Gueshooss""" "6 Whi "O008" Should "68" "moderately
<hotel, price range, moderate> <hotel, type, guesthouse>, <hotel, Internet, yes', <hotel, area, south?
Include with.
Figure 9: An example generated by CoCo with correct predictions by TRADE, SIMPLETOD and TRIPPY without retraining.
18

---

Published as a conference paper at ICLR 2021
Dialogue Flow
Turn devel Bellef
[System]: okay an
type south unit g6; cambridge leisure park, clifton road do you need
Tur
Weer] ne, but please confirm their address again and their postcode.
[Bystemj: their address is unit 96, cambridge teisure park, clifton road postoode
anything [User]: want indian food in the center area.
crestaurant, inodind crestaurant,
[System]: address unit ge, cambridge ieisure park clitton road the postcode is ebi7dy. their snything else can do for you [Gotoo] alloo looking for restaurant "alledd black" sheep"
S
crestaurant, area, northo, crestaurant, name, shoep restaurant?, crestaurant, book day, thursday>
restaurant
thursday.
Figure 10: An example generated by CoCo with incorrect predictions by TRADE, SIMPLETOD and TRIPPY without retraining.
Dialogue Flow
Dialogue-level
[System]: sure you care cheap moderately priced
[User] moderately priced in cantro.. I would like to book a table on sunday
[System]: varsity restaurant may have availability [User]: can you book a table for me?
book book 18:45>, crestaurant, food, International, crestaurant, moderates
people, Internationals, <restaurant, name, varsity restuurants, crestaurant, price range,
day, sundays, crestaurant, people, nong>, <restaurant, price range,
day, food, crestburant, range varsity
people,
original prediction
Simpist00's original prodiction
Tripyyss original prediction
Figure 11: An example from original MultWWOZ test set, which is predicted incorrectly by original TRADE, SIMPLETOD and TRIPPY, is corrected by their retraining counterparts.
Dialogue Flow
Dialogue-leve level Belief State
[System]: yes,
Turn
price range it located at 46 rose crescent would you like
reservation
[User] would restaurant centre
[System]:
food
crestaurant, ares, centrex, <reetaurant, book day, sunday", crestaurant, book people, 31>, crestaurant, book time, 10:54 pm> crestburant, food, spanish>, <restaurant, name. la raza", <restaurant, price range, cheap>
CoCo-Tums
[CoCofrara]]: can you book table for 31 people at 10:54 pm on sunday ?
<restaurant, area, centreh, <restaurant, book day. sundayo, crestaurant, book people, 31>, <restaurant, book time, none", <restaurant, food, spanisho, drestaurant, name, raza, crestaurant, price range, cheap>
<restaurant, area, contree, <restaurant, book day, sunday?, <restaurant, book people, 31>, <restaurant, book time, 19:54, <restaurant. food, spanish>, <rastaurant, name, maas crestburant, price range, cheap>
<restaurant, area, centres, <restaurant, book day, sundayo, <restaurant, book people, 31>, <reetaurant, book time, pone> <restnurant, food. spanishl, <reetsurant, name, raza?, <restaurant, price range, cheage
Trade's
Simpleroo's original prediction
Figure 12: example generated by CoCo(rare) evaluation set, which is predicted incorrectly by original TRADE, SIMPLETOD and TRIPPY, is corrected by their retraining counterparts.
19

---

Published as a conference paper at ICLR 2021
H SLOT-COMBINATION DICTIONARY
Please find the different slot-combination dictionaries introduced in the main paper below.
domain-
frng
"notel-iniee intee-iieemet"
"hotel-arez" "arra""""oe"""""""""""""""""" "'hotel-parking "botel pricerange" "hotel-stars" "hoteeltype"
hhoeel type"
"hotel area" area""," "botel internet" "hhteel parking" Baakking"""nnnn "hotel pricesange" "hotel stars"
hotellpprrknng"
"hotel-area" "hotel-intereet "bootll-priceaagge"" "hotel-stars"" "hotel-type"
hotellprioerange"
" "hotecaallolttttteeeeeee "aareeeett"""""""""""""" "botcc"pprriinnng "horkkinn"""ootstttttt "horel-starr'" /
"hotel hhotel-hookkcoo day"
("hotel book "chotlle"""oeelbbboee stay"]
hotel-ba book- people":
"hotel-book day hhote--hook stay")
book stay"
"hotel book day "houel
people"]
hotellsstas""
"botel-arca" "oaool-raa""""ottoi" "hootel-nttrree "'botel-parkingg" "hotiin"""""""pioii"i "hocc-priceenge"" "hotel-type"
hotel-area"
hhtel-internet"" ; hotel-parking" y hotel-pricerane y "hoollsttars hotel-type"
hotel-name""
"htcl-book day" bbtcl-book people" "botcl-book stay"]
"restaurant-u area"
"restaurant-foow (oodrrantt-oo""""" "oodooo"o"dd""d"aaaaaaaa"""""""" "ressuurantppiccraaee""""
"restaurani food"
"reslaurant area restaurant pricerange")
"estaauant-arca" "estaurant-ooo
restaurant
booenntt--bokkoo day" v "restaurant-boes people", "restuurresttttaanttti time]]
"cstaurant-bookk day"
restaurant-book people" "estauranttbool time"]
"restaurant bonk people"
"restaurant- book day" "restaurant- book time")
Testaurani
"restaurant book day" day""" "restaurant book people")
"uxi arrivehy"
leuvell", "rriin book people"]
Taxi-leavenl"
"aak "laxi-arriveb""" "" "train-bookk people"|
"axxi-dopattrre"
"axi-ddcttiaatio" 'axilleaveaa" "txxi-arrivbbb"]
"taxi "ossiidettnation"
("taxi "leparttre""."asse www.aiie"""ttiiiiiiiiiie
'train-arrixeby"
"train-day" "train-leavea"" "train-book people"]
"trinn departure"
"rainn arrivehy "train leavent" ", traii destination "train day train book people")
"rain-dsttiaation
"tainnarriveby" "atatnllevaa""""uiiiii departure", ("rauurre"""""idaa"""" "tran-book people")
"tainnday"
"train-arriveby"" " "rainollaveaa" : "train-book people"]
"rrain-leaveal"
("train-day")
"rain-book people"
"sttraction name""
attraction-ares"
"atraction--ypp""
attraction-type"
attraction-arca
Table 8: Slot-combination dictionary for freq case.
20

---

Published as a conference paper at ICLR 2021
slot- name
new
l'hotel-book day 'hote--namee bbotel-book stay" "houcl-priceraaee""
'hotel-interee"
'hutel stans" : 'boiel area / hhotel- book people' "aplee"""nn 'ootel type" hotel parking'1
"hotel-book day hotel-namee" hotel-book stay hhocl-priocraaaee
'hotel-area"
hhotel-star'' hotel-book people", hotel-intemet' 'hotel-rype' /hotel-parking]]]
('hotel baok day" : nomeel-nmm"" : botel
book stay /hotel procepp hhttel-stans
'hotel-parking"
'ortel-araa' hotel-book people" /hooell"hhlllaee /hhote -hotel-interee "hotel-type']
('holel Thok day 'hotel name", hotel bhotel-book stay /uote sturs", hhotel-area',
'hotet-priocrange"
"hotel hotel-book people", "oole"""oon hhhote--iteeman 'hotel-type" [botee-parkinng)
Thotel-book day" hotel-name' hotel-book stay "hooel-priccaaaee"" "'httel-araa,
'hotel-stars'
hhotel-book people" hhotel-intmmett htapeemmeethh/ttttyreee ''hotcl-tppe' ppppp"""""" "bootcl-parking]]
"hotel-book day" hotel-bookk stay "horel-pricerange" "hotel-stars" hhoters""ootel-ren
'hotel-type'
hhoellbookk people'. hotel-intemet' /hotcl-parkingg]
('hotel-book day", [ hhotel-book stay' "botel-pricennnge "hotellstarr",, "hotlssttrs.hooetareeee hhotel-aren",
"hotel-nume"
hotel-book people': hotel-intemet' h'hotel-parking""
"'htell-iamee Thotol-book stay" "hoocl-pticcraangee" /hotol-stars' Thoocl-starsshhhcccaaar "hotcl-area",
"hotel-book day'
"hutel book people' hotel intemmeee bhotel-type',Meee [botel
"hotel-book day hotel-name" hotel-book stay" "hoccl-priooranne"" "botcl-stars',
-hoke--hook people'
'hotel-area", 'hotel-interae intereet""" "hotel-type typee-yppe"" "hotel-purking"
"holel-book day" " hhtel name", : botel pricerange "hotel- stars" "aaarr"""""aas "hotel area",
"hotel-bood stay'
batek-book people", hhotel-intemet" hhoelltypee' "hotellparkingg"
I'resturratt book day "restaurant name' "restaurunt food" 'restaurant book people',
'restaurant
restaurant baok time' /restaurant- pricerange")
I'resturratt book day Irestaurant book people' "restaurant-
'restourant-fook
restaurant area" rretauurntt pricerange")
('restaurant book day /restaurant-nn name' 'restaurant-food "restaurant-bue people', restaurant-bodd time', "restaurant-
restturant-priccraage
('restaurant book day" "restaurant-hoon people", 'restaurant book time',
'restaurant-ee name"
restaurant-arca areen" aettoorant-rriccrange
('restaurant name', Trestaurant food' 'restaurant hook people', 'restaurant hook time',
'restaurant-bue bouuurant-book day"
restaurant-ure area" "restaurant pricerange" testauraat-riiceraage"
'restaurant-boo people"
restaurant- day /ostaurannt-aam /estaurant-food /estaurantt-ook time'.
"estaurant- area" 'restaurant- pricerange'
Trestaurant-bod day /rstaauanntnname /rsttarant--ood rostaurant-book people'.
'restaurant book time'
"estaurant-area" trettuurant prrturrattttrorrageee
Taass dees-ddeprrtree
duxiidestination, taxi-leaveal", "laxi-urriveby
taxl-dostination
"laxi-dopartrre" "taxillavvet" "taxi-arveby""
taxi-leaveat"
"fxxi-departuee' "ixxi-destination", 'taxi-arrivehy'
"laxi arrivehy"
lluxi departure" "lusi destination", "Laxi-leweer
'train arrivehy'
(Iriin Traiin-book people" : frriin-dyy Irainn leavent" ; 'Irain departure "train destination" ]
"train Train-leaveat
"Irain people", "train arriiebyy : "Iraindda"" 'Irain-departure" "rain destination'
'rainndcpartture
'train-book people" pcople""trainn trainnarrrieeb (tranddby "train-laaveer" htw.aaaaaannaannddddniiino "rain-destiiationn
'train destination'
('mmin-honk people'' , "train-arrivehy' 'train-day' 'train-leaveat' "rainndeparture")
train-day'
"train-book people" / 'train-rrrieby"" "rainnleavenn" "rrain-departure" "trainddestiation
'trinn book people'
"Irain arrineaariveby' 'triin 'trainddy'', 'train leaveat", 'tain departure" Irriin destination
"itraction-
"attraction-area"
attraction-arca
attraction-type'
sitraction-lype
"atraction-arraa
Table 9: Slot-combination dictionary for neu case.
21

---

Published as a conference paper at ICLR 2021
slot-nume
'hotel-intereet
"hotel-book people", hhotclboook day hhotc--aaee hotel-book stay"
hotel-area':
"hotel-book people", hhote-bbook d hhotel-name" hhotel-book stay'
"hotel parking"
[holel bank people" Chotel book day hooell name hotel book stay"
"hotel pricerange"
[hotel-book people", "hotel-book book day" notel-name"" hhotel-book stay"]
'hotel-stars"
"hotel-book phople""hotel "hotel-book day "hotel-mame" hhotel-book stay"
'hotel-yppe" hooel--name
[hotc]-book peoplc" coocc-bokkk day hotcc-booo stay'
[hotel hrotel-pricerange" : hhotelsstars' 'hotel-areal 'hotel-intermet" "hhotll-prking
poriicrange""hooelsttttt"" hotel-stars" "attsarrs"""htoorrrr""" hotel-aren" hhotel-intemeet', hotel-type" "hotel-parking
'hotel-name" hhotelppriceragee
'hotel-book day'
('hotcl-name" hamtil-nmeeehhtelheeeee hhotel-priceangee hooteegs""""ottsttrarr hotel-stars", "hotel-aren" hhotel-intemet',
'hotel-book people'
hotel-type', "hatlppinnil-
['hotel photel-priceerage""" hotel-stars' hhotel-aren", hhotel-intemee',
"hotel-book stay"
hhtel hhuee-ttppe "botel parkinng
'restaurant-boo day 'restaurant name' Trestaurant- book time'.
'restaurant-areaa
"rstaurant baok people"]
restaurant-food
[reataurant-boo day restaurant-book time". 'rsstauannt-ook people'] ]
('restaurant-bor day"; restaurantt name "restaumnnt-ood time'.
'restaurant pricerange
restaurant baok people"]
restaurant-namee
Teeaaurantt area /resturrattptiieraane""
restaurant-book day"
restaurant-name restaarannt-rraa restaurant-food restaurant- pecturant-priccaangee
'rstaurant book people'
'restaurant name' Trestaurant- area" 'restaurant food"" 'restaurant, pricerange')
restaurant-hook time'
restaurant- "rsttarant-area" 'restaurant-fodd prestaurant- /eetaarrnn--piccenne
'laxi sleputurre'
"tasi-dstiiation
texi-leavean'
"uxiidoparture" taxi-destinationn
'taxi arrivehy'
[luxi departure", "taxi destination"
'rain-arrivehy'
(trainndestinattoon.. 'rrain-epparturee
Ttain leuveal"
I'iain destination" Irain book peuple", irainn arriveby', Irain seepatture'
'rain-departure'
'rain-destination
'rain-day'
"trannddstiiatioo 'taal-doparture"
'train train-book book people'
(tmiin-rrrveeby' 'train departurrer 'train deiiindetination" 'train-day' leavent )
"altraction nume"
allractionn area"
altraction-arca
adttaccioo-niam
"ittaction-type
Table 10: Slot-combination dictionary for rare case.
22

---

Published as a conference paper at ICLR 2021
1 SLOT- VALUE DICTIONARY
Please find the different slot-value dictionaries introduced in the main paper below.
slot-nume
"notel- inveel-
yess
'hotel type"
Thotel, yuesthouse"
hottel-parking"
yes'
pooecl-proeraange:
mmodderte', 'cheap'. 'xpeesivee
("march I lith", "march 12th", "march 13th", "march 14th", "march 15th",
"hotel-book day"
"march 16th", "march 17th"""march "march 19th", "march 20th"]
"hotel-book people"
22""""""""""""""""""""""""""""""
""""""""""""""""""""""""""""""""
"hotel-hook stay"
"2""""""""""""""""""""""""""""""
hooell-aees
south', 'north" "wes", caus", 'centre"
hhoellsstars"
22, 22 34,
"hotel name""
l"moody moon", "lour
hotel", "knights inn", "travelodge",
"jack sumnmer inn". "paradise point resort")
restaurantt-u area"
"south',
"est", 'cast, "centre"
restaurant-Foo
assan fusion", 'burger, pasta", 'ramnn "aiwanese"
csstaurant-priccraage
moderatee , "chcap'. - 'expnnsive]
("huddha bowls" pouls";pizzze" pizza my heart" ""hh histro",
"restaurant-name"
"sushiya express". "rockfiie "itsl"."tsuki restaurant")
["monday". ""mesday"" "wednesday" "thursday"; "friday.
"restaurant- book day"
saturday"
'restaurant-bee people"
23
"2"""""""""" "2""""27"" "2""""""""" "'2"""
["1::01""" "8:0""""""6"""""""""" 11: 1": 19:116
19:16"""1188" "18:21"""172"""""""""" 19:31"
"18.3"""17:41" "1":4"""""""""""""""""" 11144"""1:::00 "18:51""
"7:00 pm".
"6:07 pm11 pm "7:11 pm", "6:17 pmn"."5:77 pm"
"restaurant-bokk time":
pm3",";37 pm "S:42 pm 77:47 pan" "6:52 pm"
"5:57 pm" 111:00 am"" 11:05 "1="""111100011 um"."'"""""""" am",
"11:20 am"""1 "m1:30 am am","11:35 am" 11:40 am".
"11:45 am" 1:55: "11:55
"7:00 pm"""6:07 pm "S:12 pm 77117 "617 pm"
"taxi-arriveby"
"5:27 pmp""I 1::0 am"." 11:35 am" am""""11:40 "m"""""m""""" 1::45 am",
"11:50 am". am")
"19:01". "1::6"""1:111""""""" "1..1"""1 "1:22""" pmi"
"6:37 "5:42 pm "7::4 pm pmm5"":522ppppp" pen".
"taxi leaveat"
pm 11:00 "m" 11:05 am" am"?"11:100
"11:15 am" 1::2 am" am."1.mmm"111..0 11:25 am"]
("moody moon". "four seasons hotel", "knights inn",
Tuxi-departuree:
"travelodge", "jack summer inn". "paradise point resort"].
"buddha bowls"."pizza
licart.."poa bistro".
"aaxi-datiiation::
"sushiya express" "rockkire grill" "itsuki restaurant"
"".""""""""""""""""""""""""""""" 18:35" 7770 "6607 pmm1" pma'.":17 pm".
"train-arriveh":;
pmi"."::77 pm pm""11:30
"Im4""11:0 am".
"11:45 am" 1:50 am" 11:5 am")
["1::01 (18:0""" 17:1 1":""""""""" 19:16""""""""""""" 19:16""118211 18:21" "7:32 pm",
"train-lcaveat":
"6:37 pm"""":42 pm "m="""""7""""""""""""""""""""" pm"" 76:52 "mm5" pm"
"11:00 "1:05 am am" "11:10 am"."11:15 am" "I 11::5 am"l
['ailroy" san martin morgan bill" 'boosoo hill".
"train deniindeparture"
"college park"."sante clara" "lawrence""" "'uuny
I"mountann view" sun antonio" ". puulo allo" "ooo"""""ee "menlo park".
"train destination"
"hayward park san
"broadway" "saan hruno"]
I'march 1ith". "march 12th". "march 13th", "march 14th", "march 15th".
"train day":
"march 16th", "march 17th" ""marhh 18th", "march 19th", "march 20th"
book people"
"22" "24" 25"""22"" "2"""""""""""""""""""""""""""""" 2877""88"""""""
"sitraction area"
south', 'porth', "west", 'eust' "centre"]
I" "grand canyon" golddn gate bridge" ''naagrra falls".
"sttraction-aame"
"kennedy space center", "aaleerr""""pp"""e "poke place market" "Tlas vegus strip"]
aatraction-type"
"historical landmark". 'aquaria" "beach", "castle" mostlo"/art gallery
Table 11: Slot value dictionary of train-O.
23

---

Published as a conference paper at ICLR 2021
slot-name
hhoellinntrmet"
l'yes'
"hotel-type"
[hotel", "guesthouse"
"notel- purtel-prkingg
Iees'T
"hotel pricerange
mmoderatee ; 'cheap'. 'expensive"
"notel-book day"
"Inday,, "wesdayy. "Bursdayy saturday,, mmndayy" sundayy, 'eddessay
'hotel-book people"
[T,, 2, 4,,4,,3 4..3 77.81
"hotel hook stay"
[T', 2", 33, 14,5", '7.87
['alpha milton,, m'lloonmillon,
"flinches bed and breakfast', 'express holiday inn by cambridge', 'wankworth house". 'alexander and b". 'the gonville botel']
"hotel-name"
"hotel-stars"
"O',"', "3", 14,,""""""
"notel-ares"
'south", "east" 'west., 'morth. 'centre"
"restaurant area"
'suuth, 'east,, "west", 'norll", "cenire")
"restaurant- food"
'europeon', 'bradlian', 'weish')
"cstaarant-priccaange"
moderaase., 'heapp, 'expensive")
['pizza cherry, 'the nimla', "barbakan', 'the golden house', 'michaelhouse""
"restaurant name""
'bridge., 'varsity restaurant" 'locch, 'the peking', "charlis', 'cambridge lodge'.
"maharajah tandoori")
"restaurant- book day"
Irday', , "wuedday' "Burrdayy" , 'saturday', "monday". 'sundayy, 'wednesday"
"cstaauantt-ook people"
['8', 6.
'. ",, "T, 33, '2, (3]
"restaurant-e bonk time"
[11:40', '19:00', 115:15" '9:30', '7 pm", am'; '8:45"]
"axii-arrveby"
108.30,, 9::55]
"tuxi leaveat"
['7 pm' "3:00"]
aylesbray lodge". 'lizzillies", 'uno", "zizzi cumbridge", "express by holiday inn'.
great saint marys church', "county folk museumi' "riverboul" 'bishops stortford",
"taxi-doparture"
caffee uno', 'hong house', 'gandhi', "cambridge arts', 'the horpot", 'regency gallery'.
'saint johns chop shop house') ].
Tashleyy. 'all sainss', : "do cucina and bar's", . lensticeld botel", bistro". 'brosbourne" 'sleeperz hotel', "saini catherine's 's cullege")
"taxi- destination"
['4:45 pmi '18:35. 21108"" "I9:54" '1008", '13:06", 13:24". '0708", "16:23. '8:66.
'09:01', '10:23', '10:00 um" '16:44". '6:15', "06:01", "8:54"/1::51" "16:07', "12:43'. '20:08,, '08-23', '12:56', '17:23", "11:32", "20:54", '20:06', "14:24", "18:10", '20:38",
"train-arrvvoby"
'16:06' '3:00', '22:06', 220:20', 119:51" "7:52', '07:44, '16:08"]
['13336, 15:17", (1::11, 13:15 pm 6610 am '14:40,, 55:40" "1.T"," 13:30
"5:11", '11:17", '5:01", '13:24", '5:35', '07:00", "8:08", '7:40', "11:54', '12:06'. '07:01',
"train leaveall
'18:99" '13:17', '21:45", '06:40" "01:44", '9:17', '20:21', '20:40', "08:11", '07:35', 114:19; 'I pm", "19:17", "19:48, "19:50" 100:36, '09:19", "19:35', '8:06', '05:29", 117:50,, 16, '09:17', '7:35', '5:29", '17:16,, '14:01", '10:21', '05:01', 115:3', '15:01, '10:11, (0::01).
['london liverpool street', 'iings lynn', 'norwich", ; 'birmingham new street.
"train-departure::
'Kondon kings cross', mouss"""braaa /broxboume""
bishhops storiford cambridge" cly". stansted airport. peterborough
leicester'.
"train-dcatination""
'tevenage"]
"train-day"
"Iidayy. ; tuesday', "thusday", mmodday, 'saturday" sunday", 'wodnesday"
"train-bokk people"
[y]
['tee cambridge arts thentre', 'the churchill college', 'tee castle galleries", 'cambridge", "saint catherinc's 's college", 'strect', 'com cambridge exchange". 'fizwillia''. 'cafe jello museum") I.
"attraction-namee.:
attractionnarcaa:
|south, ceas', 'west,, "north"; "centre)) ).
"sttraction-type"
concerthall" "muscum", ensertainment' college', , 'multiple spotts', 'hiking'. 'urchitecture', 'theatre', 'cinema", 'swimmingpool: 'boal', 'nightclb'; 'park'1
Table 12: Slot-value dictionary for case.
24

---

Published as a conference paper at ICLR 2021
slot-name
hhotel-intemmt""
[yee]]
"notel-lype"
[hotel", "uesthouse"
hotel-pprrking"
I'yes'
"holcl-pricerange"
"moderate", "cheup". "expensive"
appiil Hith "april 12th "apeil 13t6", apri 14th april 15th
"hotel-book day"
"april 16th", "spril 170"" 18th", "april 19t1". "uprill 200h"|
hhoeelbbook people"
""30"""1" 52""1"3 333"""34""" "33""""7"""" "391
"hotel-hood stay"
"3""""""""""""""""""""""""""""""
hhotel-ares
''outh"; "east" 'west', 'north', "centre")
"hotel stars"
[O'' 'O, '2", 13', 14"
('white rock hotel", "jade bay resort", "grand hyait", "hilton garden inn"
"hotel-name"
"vollage motel"" "mandarin oriental"].
"restaurant- area"
''outh", 'east,, 'west", "north', "centre']
"restaurant- food""
"sscchun"" "lish", "noodle", "lobster", "burrito", "dumpling". "curry".
"cstaarant-rriccaagee
moderaaee, 'chaap.. 'expensive]]
["lure house" "aause""""haa"" '"llack sheep restaurant" " "palapa restaurant", "nikka rimen",
"resaauaant-aane"
"sun sushi" ''super cucas")
"rstaaranst-bod day". "
"mondyy" "Iuesday : "wednosday" , "thursday" "Iriday" "'saturday" "sunday"
'esttarant-bonk people"
"31""""""""""""""""""""""""""""" "3"""""""""""""""""""""""""""""" "36", ""3""""""""""
"20:02", 22::""22222""""""""""""" "2" "21::22""""""""""""""""""""""""" "22:27""0:322"222::1
pm"""9 "9:04 pm"."10:09 10:09 pm" pm"
"restaurant baok time"
19:1 11:24 pm"2"8:29 pm""":34 pm" pmn"""0:30""""""""""0000"0"""""" pm" 78:44 "9:4"":49 pm".
110:54 pm=""10:00 am""""0:00 am" 110:11 10066 am02" "10:61 am",
110:31 am ' 101033 am". "10:4 am" "10:4 am "10:51 am". "1055 am"6"" "].
pm"."10:39 pm".
"tuxi-arriveh";;
"8:44 pm """ 99:44 pm" "10:54 pm". "10:00 am "." 10:06
am","10:1 ",10: am".
"10:16 11:21 "10:66 am"].
["21:37" ""::""""""""4""""""""""""""""""" "0"4"""""""""""""""""""""""""""" "8:00 pm "9:04 "1009 pm",
"taxi leaveal""
"8:14 "9:19 pm "110:24 pmn "8:29 pm 10:31 am" "10:36 am", "I0:4] am".
"10:46 100:1 "1:500 am"].
("lure lish house" "ouuse""""aa" "black sheep resiaurant" " palaapa restaurant", " "mikka ramen".
"taxi departure":
sushi"" 'super eucas"],
('while rock hole!", "jude buy resort", "grand hyait", "hillon garden inn".
"taxi-dctiiationn:: :
"collage "maaddrrnanrriii oriental"]
I"northrdgee" ""camaiilll"""" ""xxaard" ; "morepark"" " "simi valley" "ollley""""ha"a "chatswonn""
"train-departure"
"van "'lendall"
["norwalk"" "hheen park"""ullleton"" " '"kanta ana" ""tustin" ". ""irvine",
"rainnddstinaatonn"
"lemente"""cccmmaee""" '"oceansidc"].
220002"211:0 "22:12 "20:1" 20:17""21122" ""2:277" 19:34 pm 10039 pm"
"train :
"8:44 pm""." pmn "110:54 pmn" "100000:00 am" am""" 110:06 am". "10:11 am".
"10:16 "1021 am""10:26 am"].
''apill 1ith", "april 12th", "apeil 13th", "spril 140"", "april 15th".
"train-day":
"april "april 77dh"/'apri 18th", "april
"spril 20th"],
78:00 MM." 1mn""10:99 pmn". "8:14 pm"; "9:19 pm" 110:24 pm pm"" "8:29 pm" " 10:31 am""""0:36" 10:36 am",
"21:37".2244 "2:4"""2
"train-lcaveat":
"10:41 10::4 "10:51 am". "10:6 am").
"train-book people":
"30""""""""""""3"""""""""""""""" """""""""""""""""""""""""""""""" """""""""""""""""""""""""""""""" "37"""""""""""""""""""""""""""""
"attraction area".
'souh', 'east", 'wes',, 'morth' "centre") ]
("statue liberty". ". "empise
building : ("mount rushmmoee".
"attraction-name":
"brooklyn bridge","Tincol" memorial" "times square"].
"attraction :yrrctiion-tpee::
("temple", "200", "libarr", "skyscraper" " "monumen""
Table 13: Slot-value dictionary for O case.
25


Reviews:
[{"review_id": "SwEZ-ctRzQN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "<Summary>\n\nThis paper addresses the problem of evaluating dialogue state trackers (DST)’s generalization ability to novel and realistic dialogue scenarios that do not exist in the dataset. It proposes a model-independent approach to evaluate DST systems with the idea of counterfactual conversation generation. The proposed approach is integrated with three recent DST models and evaluated on MultiWoZ dataset. \n\n<Strengths> \n\n1. The idea of counterfactual goal/conversation generation can be useful for the evaluation of DST systems, which is often quite challenging in practice. \n\n2. The proposed CoCo idea is tested with three recent DST systems including Trad, TripPy and SimpleTod and show its effectiveness on MultiWoZ dataset. \n\n<Weakness>\n\n1. This paper proposes an interesting idea for DST evaluation but its implementation is largely ad-hoc and engineering intensive and thus bears little technical novelty.\n\n(1) As described in section 4 and Fig.2, the proposed approach is simply a combination of multiple components with heuristic rules (e.g. the way of value substitution, and the use of some predefined operations). \n\n(2) The two filtering schemes are also ad-hoc, consisting of two slot-value match filter and classifier fitter.\n\n(3) It is hard to find methodological novelty in the proposed method. Given that ICLR is a top premier ML venue, it could be a significant weakness to be a publishable work. Thus, this work may be better fit to a venue of NLP.\n\n2. Experimental evaluation can be improved. \n\n(1) This work evaluates only on a single dataset – MultiWoZ 2.1. Admittedly, it is one of the most important benchmarks for DST tests, but one or more datasets are encouraged to use for better justification of generality, for example, MultiWoZ 2.0 or Taskmaster-1. \n\n(2) The experiments mainly focus on how much the proposed CoCo improves the three DST systems. However, comparative study is required such as how much CoCo is better than other alternatives or baselines. \n\n<Conclusion>\n\nMy initial decision is ‘reject’ mainly due to lack of technical novelty. Experiments could be improved for better evaluation. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JYWGFYHmsr5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting approach to generate dialogs in a controllable fashion to evaluate a Dialog State Tracking system on a data distribution which is different from the training/test data. The proposed approach first generates a turn-level goal by adding or dropping a slot and then replacing slot values. In the second step, the proposed method generates counterfactual conversation conditioned on the dialog history and goal generated in the previous step. The authors show that evaluating current state-of-the-art DST model on MultiWOZ datasets with the generated counterfactuals results in significant performance drop. Additionally, human evaluation shows that the generated conversations perfectly reflect the underlying user goal. \n\nThe paper is trying to tackle an important problem of evaluating robustness of a DST model when most of the available datasets has similar distribution in train and test splits. The proposed method is well explained and the effectiveness of the approach is substantiated by extensive results.\n\nIt is not clear what is the performance of utterance generation model. This model is somewhat different from other language model since it is conditioned on belief as well. Also, seems like the proposed approach can generate counterfactual conversation only for one turn which seems limited for the evaluation.\n\nI would like to hear from authors on:\n- In section 5.4, it is not clear if slot combination dictionary was also split such that slot combination in train and test data doesn’t overlap? If not, model can learn just the generated utterance pattern and perform better on the generated test set. \n- Performance of the utterance generation model in terms of diversity of the generated utterances since it will affect the evaluation’s robustness. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "CvZ6d0LROU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after reading author response:\n\nThank you for the thoughtful response.  I appreciate that you have added extra implementation details and am changing my score to a 6.  \n\nRegarding the limitations in scope:  My apologies if my review was confusingly worded.  I just wanted to clarify that I had meant that the counterfactual generation method itself may have limitations (not the high-level DST task, which I agree has broad uses).  The concern is that the adversarial example generation strategy might be too domain-specific to transfer easily to other tasks, and that this might limit the impact of the proposed counterfactual generation method.  I think this is somewhat related to the parts of the methodology that R1 described as \"ad-hoc\", \"engineering intensive\", or reliant on heuristics.  I still have some reservations about the transferability of the proposed methods, though the authors' response to R1 did clarify a bit on this point.\n\n-------------------------------------------------------------------------------------------------------------------------------------------\n\nOriginal Review:\n\nThis paper presents CoCo, a new method for generating adversarial examples for DST tasks using “controllable counterfactual” generation.  Unlike other approaches for adversarial example generation, this approach is model agnostic.  They also demonstrate the effectiveness of the method by applying the Multi-woz dataset where s.o.t.a. Model performance drops by ~30 points.\n\nI might lean towards rejecting this paper because there are several points in the methods that are still unclear to me.  However, I hope the authors may be able to clarify some of my questions (I’ve listed a few in the limitations section of my review).  Another potential drawback is that the methods described here are limited in scope to the dst domain, so it may not be as useful to other ML researchers more broadly.  \n\nStrengths:\n- CoCo is a very effective method.  Model performance drops by about 30 points, demonstrating that the adversarial examples generated are extremely challenging. \n- The human study demonstrates that the created examples are still human-like and are actually often rated “more correct” than the original Multi-Woz responses.\n\nLimitations:\n- The proposed methods are using a lot of pre-existing components and limited in scope to this one domain.  While the created counterfactuals are useful as a challenge dataset for DST systems, the overall approach in this paper may not be more broadly impactful outside of this task.\n- There are a few points in the methodology that seem a bit unclear or should be described more fully:\n  - Can you provide more details about the conditional generation model (p):  How is it being trained?  What architecture is being used?\n  - The Slot-Value match filter: How are you judging whether the candidate “contains” the value? (Is this exact match?)\n  - Classifier filter:  Can you provide more details on the classifier architecture and how it is being trained?  What is the precision and recall of its predictions?\n\nQuestions:\n- In section 5.4: Any thoughts on why data augmentation is more effective for the Trippy model than the other two models?\n\nMinor Feedback:  \n- The CoCo name could be easily confused with Microsoft Coco, a commonly used computer vision dataset.  Authors might consider changing the name to avoid confusion.\n- The font in Figures 3 and 4 is tiny and hard to read.  I recommend making it bigger.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method  to generate conversations for evaluate dialog systems using counterfactual generation. 

Pros:
- The reviewers agree that the paper makes a good contribution towards evaluation of DST models. 
- The paper adds to a growing body of work on robust evaluation of NLP models

Cons:
- One reviewer has commented on the lack of novelty. However, I believe that the authors have adequately addressed it. In particular, I do not see any harm in using heuristics/templates to generate counterfactuals as long as the final goal of robust evaluation is achieved. 
- It would have been good to evaluate the method on other datasets. However, I agree with the authors' rebuttal that this is indeed the most popular dataset for the task and most SOTA methods evaluate on this dataset. 

The authors have adequately addressed all reviewer concerns and have clearly highlighted their contributions and novelty.

I think of this as a valuable contribution and would like to see the paper accepted. 
Decision: Accept (Poster)