Building Multi-domain Dialog State Trackers from Single-domain Dialogs
Authors: Qi Zhu
Zheng Zhang
Xiaoyan Zhu
Minlie Huang
Venue: EMNLP 2023 Main (2023)
Keywords: dialog state tracking
multi-domain dialog
conversational query rewrite

Abstract: Existing multi-domain dialog state tracking (DST) models are developed based on multi-domain dialogs, which require significant manual effort to define domain relations and collect data. This process can be challenging and expensive, particularly when numerous domains are involved. In this paper, we propose a divide-and-conquer (DAC) DST paradigm and a multi-domain dialog synthesis framework, which makes building multi-domain DST models from single-domain dialogs possible. The DAC paradigm segments a multi-domain dialog into multiple single-domain dialogs for DST, which makes models generalize better on dialogs involving unseen domain combinations. The multi-domain dialog synthesis framework merges several potentially related single-domain dialogs into one multi-domain dialog and modifies the dialog to simulate domain relations. The synthesized dialogs can help DST models capture the value transfer between domains. Experiments with three representative DST models on two datasets demonstrate the effectiveness of our proposed DAC paradigm and data synthesis framework.

Building Multi-domain Dialog State Trackers from Single-domain Dialogs
Qi Zhu, Zheng Zhang, Xiaoyan Zhu, Minlie Huang"
The CoAI Group, DCST, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems,
Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China zhu-q18 maliigl@@@mill..em wiigguiissingaa.eeuuu. axads.,,aand @ssinghua.edu.cm
Abstract
Bus domain
Usr: want bus tickets!
Existing multi-domain dialog state tracking (DST) models are developed based on multi- domain dialogs, which require significant man- ual effort to define domain relations and col- lect data. This process can be challenging and expensive, particularly when numerous do- mains are involved. In this paper, we propose divide-and-conquer (DAC) DST paradigm and a
Sys: Where are you going and what time do you plan leave?
Usr: want to leave on the 3rd March at 12:45. want to visit San Francisco.
...(few turns later)...
Usr: Sounds fantastic.
Sys: Your reservation is confirmed.
State: [(Bus, departure_ 3rd of March), (Bus, to city, San Francisco)...]
Hotel domain
dialog synthesis framework, which makes building multi-domain DST mod- els from single-domain dialogs possible. The DAC paradigm segments multi-domain di- alog into multiple single-domain dialogs for DST, which makes models generalize better on dialogs involving unseen domain combi- nations. multi-domain dialog synthesis framework merges several potentially related single-domain dialogs into one multi-domain dialog and modifies the dialog to simulate main relations. The synthesized dialogs can help DST models capture the value transfer between domains. Experiments with three rep- resentative DST models two datasets demon- strate the effectiveness of our proposed DAC paradigm and data synthesis framework.
Usr: Could you search for hotels that area as well?
State Update: [(Hotel, location, San Francisco)] Sys: Sure. What is your date of checking in?
Usr: On the same day.
State Update: [(Hotel, check_in iate, 3rd of March)]
Table An example of cross-domain value transfer. The values of location and check_ in date slots in the Hotel domain are transferred from the Bus domain.
To model the domain relations in multi-doman dialogs, previous works either employ a copy mech- anism (Heck et al., 2020; Jiao al., 2022) or en- code a pre-defined schema graph that indicates slot relationships using prior knowledge (Zhu et al., 2020; Chen et al., 2020; Feng et al., 2022). Nev- ertheless, these models are data-driven, relying on adequate multi-domain dialog data. Another line of research is conversational query rewriting (CoQR), which rewrites user query to be self-contained and understandable without the dialog context (Su et al., 2019; Rastogi al., 2019; Quan al., 2019). However, these methods also require manually col- lecting in-domain query rewriting data.
1 Introduction
recent years, with the emergence of multi- domain dialog datasets (Budzianowski et al., 2018; Rastogi et al., 2020), the focus of task-oriented dia- log (TOD) research has shifted from single-domain dialog multi-domain dialog. In multi-domain dialog, there may be relations between domains, often exhibited through the transfer of slots' values as illustrated Table 1. However, as the number of domains increases, manually defining domain relations and gathering multi-domain dialog data becomes increasingly expensive. In this paper, we investigate how to build multi-domain DST models from single-don sinleedoomin dialogs, which may help scale TOD systems to handle numerous domains.
When there no prior knowledge about the data distribution of multi-domain dialog, building DST models faces two main challenges: (1) how to make models generalize well to multi-domain dialogs involving arbitrary domains, (2) how to en- able models to capture domain relations. In this work, we propose a divide-and-conquer (DAC) DST paradigm and multi-domain dialog syn- thesis framework address the challenges. The DAC paradigm segment a multi- dialog into single-domaie dialogs and perform DST for each
"Corresponding authors.

---

domain separately, bridging the data distribution gap between dialogs involving different domains. DAC, models only need to predict the state of the currently active domain according to the dia- log history of that domain and the state of other domains, which greatly saves the computation of reading all dialog history and predicting all do- mains' states. Our proposed data synthesis frame- work combines multiple single-domain dialogs into a multi-domain dialog and simulates value transfer between slots of different domains. The frame- work includes (1) identifying candidate slot pairs for cross-domale domain value transfer, (2) concatenating single-domain dialogs based on candidate slot pairs, (3) rewriting utterances to simulate cross- cross-domain value transfer, and (4) filtering problematic rewrites with value tagger. The synthesized data can be used to train DST models directly or train CoQR model to resolve cross-domain value transfer be- fore DST. Experiments on two datasets with three representative DST models show that our proposed DAC paradigm and data synthesis framework sig- nificantly improve model performance on multi- domain dialogs when only single-domain dialogs are available.
the value from the dialog (Zhang et al., 2020; Heck et al., 2020), and directly generating the value (Wu et al., 2019; Hosseini-Asi et al., 2020). Value pre- diction can be executed at either turn-level, where models accumulate the predictions of each turn to obtain the complete state (Kim et al., 2020; Lin et al., 2020), or dialog-level, where models pre- dict the complete state directly (Hosseini-As)) et al., 2020; Peng et al., 2021). In this paper, we verify the effectiveness of our propose DAC paradigm and multi-domain dialog synthesis framework on both turn-level and dialog-level generative DST models.
In multi-domain dialog, slots from different domains can be correlated. The slot relationships can be modeled implicitly through self-attention or graph neural networks (Kim et al., 2020; Ye et al., 2021; Lin al., 2021a), or explicitly through copy mechanism or schema graph that encodes prior knowledge (Heck et al., 2020; Jiao et al., 2022; Zhu et al., 2020; Chen et al., 2020; Feng et al., 2022). However, all of these methods are driven by multi-domain data. When multi-domain data are unavailable or insufficient, our proposed method provides valuable solution.
In summary, our contributions include:
2.2 Multi-domain Dialog Synthesis
1. We propose a divide-and-conguer paradigm for DST, which improves model efficiency and gen- eralization on multi-domain dialogs.
To reduce the cost of collecting multi-domain di- alogs, some works propose synthesizing them from limited number of et al., 2021; Mohapatra al., 2021; Wan et al., 2022) by fine-tuning pre-trained models to gener- ate new dialogs based on randomly sampled user goals. On the other hand, Li et al. (2021) rewrite the user query according to counterfactual user goal in order to assess the robustness of DST models to unseen slot values and rare slot combinations. Dif- ferent from these works that assume the possible domain dependencies are known and real multi- domain dialogs are available, we identify potential domain dependencies and build models using only single-domain dialogs.
dialogs (Kim
2. We propose data synthesis framework that can generate multi-domain dialogs using only single- domain dialogs, which enables DST models to identify domain relations without annotations. 3. Experiments show that our proposed methods can substantially boost multi-domain DST mod- els using single-domann dialogs, which reduces the need to collect real multi-domain dialogs.
2 Related Work
DST
2.1 Multi-domain Dialog State Tracking
Dialog State Tracking (DST) is a key module of TOD systems, aiming at capturing the user goal in the form of (domain, slot, value) triplets dur- ing a conversation. The introduction of large-scale multi-domain dialog datasets, such as MultiWOZ (Budzianowski et al., 2018) and SGD (Rastogi al., 2020), has led to the development of various multi-domain DST models. These models employ different techniques for predicting the value of a slot, such as selecting the value from a pre-defined set (Lee et al., 2019; Chen et al., 2020), extracting
3 Background
3.1 Notation
Let C. S1,, Ut] denote a dialog until the t-th turn, where U, and S, are user and sys- tem utterance respectively. The dialog state B, {dd, s, N. consists of Ne (domain, slot, value) triplets expressed in C., where the value is not empty and there is only one value for each domain- slot combination. Following Lin et al. (2020), we

---

denote the state update from B, 1 to B₁ Lt. Lt has similar format as B, but may contain slots with empty values, which indicates that values from B--1 are removed in the f-th turn.
Bus
Bus
B Bus
CH
Hotel
B Hotel
We denote the active domains of turn as Dj- If Le is empty, De is the same as Dz Otherwise, Di consists of the domains in L, Fort to < t' where d' is the first turn that Ly is not empty, we set D, domains in Le. turn is regarded as a eross- domain (CD) turn if there exists a triplet (d, 8, v) in L,, such that (1) 0 is not empty, (2) e appears in previous turns instead of the current turn, and (3) the domain d is not in the active domains of the turns appears. We define such behavior as cross- domain value transfer. When dialog has cross- domain turns, it is called cross-domain dialog.
Resst
Restaurant
BRest
Dialog
State
Figure I: Divide-and-conquer DST paradigm.
4.1 Divide-and-Conqee DST Paradigm
As illustrated in Figure 1, in the divide-and -adiiide-nd-oouuuerr (DAC) DST paradigm, a multi-domain dialog is segmented into single-domain dialogs, then the DST model predicts the state of each domain sep- aratcly. Formally, for each turn active domains D, using domain classifier (Dom- CLS). In most turns, De contains only one domain. Thus, a multi-domain dialog can be split into sev- eral single-domain dialogs. For each d E Dr, the DST model predicts the domain state B3 according to domain context Cf and previous other domains' state B) where Cy Cr is the first turn that appears in Dr By updating previous state B with {B;') ] we get the current state Bi. Compared with existing dialog-level DST mod- els that predict B, from C₁ directly (e.g.. T5-DST and SDP-DST), the DAC paradigm has the follow- ing advantages:
identify the
3.2 Generative DST Models
this work, we use generative DST models that generate the values of slots, as they are flexible and require minimal annotation. Specifically, we consider three representative models:
T5-DST (Zhu et al., 2022) predicts the serialized state B, from the entire dialog context C..
MinTL (Lin et al., 2020) predicts the serialized state update L, from the partial context C wet and the previous state B where w is the con- text window size. We set w to two to include two previous turns. During inference, we input pre- dicted previous state B/_1 instead of B, and update By to B/ using L/-
The model always takes single-domain dialogs as input, bridging the data distribution gap between dialogs involving varying domain combinations.
SDP-DST (Lee et al., 2021) predicts the value of each slot independently with entire dialog context Ct. domain name and description, slot name and description, and example values (if any) as input. We use T5-Large (Raffel et al., 2020) as the back- bone for all models. For MinTL and T5-DST, we use the same state serialization function provided by ConvLab-3 (Zhu al., 2022).
The model takes concise state B instead of other domains' dialogs as input, saving the com- putation and facilitating copying value from other domains.
Besides, the DAC paradigm can greatly improve the efficiency of models that predict the value of each slot independently (e.g., SDP-DST), particu- larly when there are many domains. the DAC paradigm, these models only need to predict the state of the active domains instead of all domains.
4 Method
Unlike previous works on multi-domain DST, we assume we only have single-domain dialogs with no access to prior knowledge regarding how do- mains will correlate in a multi-domain dialog. In this setting, DST models face two challenges: (1) how generalize dialogs involving multiple do- mains, (2) how to model the domain dependencies. We propose divide-and-conquer DST paradigm and a multi-domain dialog synthesis framework to address the two challenges.
For turn-level DST models like MinTL that pre- diet state update L, from partial context Ci-wa with fixed window size w, the DAC paradigm provides more complete context for domains in L4, According the definition, for domain d in I,, Od contains all turns that d is active.
We apply the DAC paradigm to TS-DST, MinTL, and SDP-DST models. The domain classifier Dom-

---

C
[user] I want bus tickets! [user] On the same day.
mation or contain redundant information. The synthesized multi-domain dialogs can used to train DST models directly or to train CoQR model to rewrite utterances before DST for resolv- ing implicit cross-domain value transfer.
Ca-2a
[user] Sounds fantastic. [user] On the same day.
Chocel
[user] Could you scarch for hotels in that area as well? [user] On the same day. [Hotel]
Dj
B Bi
[[ts][[to city][San Francisco], [departure ]
4.2.1 Slot Relation Mining
[3rd of March ...)
B
[Hote][[[ocatioo][Sa] Franciscol)
The initial step in synthesizing multi-domain di- alogs is identifying the relations between slots from different domains. Given single-domain dialog of domain d, our objective is to identify target slots in other domains that accept the same value non-empty source slot in d. For example, given a dialog in the Bus domain, the location slot (target) in the Hotel domain may have the same value as the to_city slot (source) in the Bus domain.
B Hotet
H HC-1
Motel
B
[[chek][[heckk in [are]]]rdd of March), [loca-
[ion][San Franciscol)
B
B Hotel Blottel
B:
B Bo Bfoteel
Lt, L.fotel
Beaicchcckiinddane [chce]][
3rd of March))
X(dm,8n 8n)
[domain] Name(dm.) Description(da)) [slot]
Name(s..) Description(s..) ([PVs] 01, ...)
TS-DST
C. (B) C Chooee, D.) + Hifutel
DAC
Inspired by TransferQA (Lin et al., 2021b), we formulate DST Question Answering task to enable zero-shot cross- doos-d inference, where the input is "what is the (slot description) of the [domain name) domain? [dialog context)" and the output is the slot's value. Initialized by UnifiedQA- v2 (Khashabi et al., 2022), our QADST model is trained on single-domain dialogs. For a dialog of domain d, the model is trained to predict the values of slots in d. When performing cross-domain infer- ence, the model predicts the values of target slots in other domains. By comparing the predicted target slot's value and the source slot's value, we further filter out slot pairs (source slot, target slot) that are generally less consistent (Fl score lower than 0.1), For each single-domain dialog, we record the target slots predicted to have the same value as a non-empty source slot in the current domain.
MinTL
(Ci-2a; BL-1) Lt (B) Offocet, Bilotel, Di) + L Hotel
DAC
SDP-DST
(Ce: X (dm: 8n)) (B) , ( en)) oi
v/"
Hote! Table 2: Input and output of DST models with/without
DAC
DAC paradigm for the example dialog in Table 1.
CLS T5-Large model that takes the last two turns Ct 2:t as input and predicts active domains Dr. Table shows the example input and output. In the DAC paradigm. T5-DST and SDP-DST only need to predict the state of the current domain (Hotel), while dialog contexts of other domains (Bus) are replaced by their states. MinTL with DAC uses the turns of the current domain input instead of the last two turns. The DAC paradigm could also be applied other kinds of DST models.
4.2 Multi-domain Dialog Synthesis Framework
4.2.2 Concatenation and Value Replacement
We combine multiple single-domain dialogs into dialog. While we illustrate the concatenation of two single-domain dialogs for of explanation, our method be readily extended concatenating multiple single-domain dialogs by iterative appending.
Previous DST models learn domain dependencies from multi-domain dialogs. Instead of manually collecting multi-domain dialogs, we propose data synthesis framework to generate multi-domail di- alogs from dingledommin dialogs automatically. As illustrated in Figure 2, the framework includes the following processes:
We randomly sample the first dialog for concate- nation. According to the annotation of target slots in the first dialog, the second dialog is sampled from dialogs with at least one non-empty target slot. To improve the fluency of the concatenated dialog, remove the last turn of the first dialog if has an empty L,, as this is typically indication that the user is ending the conversation. For the second dialog, we substitute the target slots" values in the utterances and states with the correspond-
1. Mine the slot pairs that could potentially take the same value in specific dialog context.
2. Concatenate single-domain dialogs that contain relevant slots. Replace target slots' values with source slots' values.
Rewrite user queries in the concatenated dialog to implicitly express the value of target slots.
4. Filter out the rewritten queries that miss infor-

---

Bus
SFO
Bus
Dialog
Portiand
Cross-domain Value trunsfer
Concatenation
Utterance Rewrite with CoQR model
Filter Value Tagger
Single-domain
Value Replacement
Dialogs
Hotel
Dialog
Synthesized Multi-dommin Dialogs
An illustration of proposed data synthesis framework. The graphs near the dialogs are their final dialog state, where the centric nodes are domains and the green nodes are slots. In this example, the value Bus's s to_ to_city slot ("SFO") transfers Hotel's location slot. More single-domain dialogs be merged into multi-domain dialog by repeating the dialog concatenation, value replacement, and utterance rewriting processes.
Figure 2:
Could you search for hotels in Portland? Value Replacement: Could you search for hotels in San Francisco?
Original:
slots. As shown in Table 3, the spans of target slots' values in U, are labeled with substitution operation tokens. To provide sufficient information for rewrit- ing, the context C consists of the descriptions of the corresponding source slots and their contexts, along with the previous system utterance S. Formally, C [Cn Vi, Ct2 1:ty, Ctg-1d2 VeN S where 4 is the first turn that the i-th source slot-value appears in It.,, Ct,-1:t = 11,-1, Uv.) and Ve, is constructed by filling template "The [slot description | is (value)".
Utterance Rewrite:
Input: Could you search for hotels in <sub> San Francisco </sub>? ?
Output: Could you search for hotels in that area?
Table Illustration of value replacement and utterance rewriting.
ing source slot values from the first dialog. When appending more dialogs, the subsequent dialogs are processed similarly to the second dialog, ex- cept that their target slots can refer to slots of any previous dialogs.
4.2.4 Filter by Value Tagger
We further filter the noisy rewritten queries to en- sure they do not miss information or contain re- dundant information. We employ a value tagger to extract the values of the original user query and rewritten one. The rewritten query must not con- tain the values of target slots and must only contain other values in the original query. The tagger is RoBERTa--arge (Zhuang et al., 2021) with a ken classification head, which assigns one of threc tags (B, O} to each token. the Taskmaster datasets (Byme et al., 2019, 2021) utilizing dialog acts annotation.
4.2.3 Utterance Rewriting
For the turns in which the state update L, includes target slots for the first time, the user utterances further rewritten to express target slots im- plicitly, simulating cross-domain value transfer be- havior. Instead of manually collecting in-domain query rewriting data for training, we train CoQR model (CoQR-R) on CANARD (Elgohary et al., 2019) dataset, which contains conversational ques- tions Uromer and their self-contained rewrites Use Unlike Elgohary et al. (2019), our model takes the context C and rewrites U as input to predict the original question (romm that may contain anaphora and ellipsis. Since our goal is express the tar- get slot's value implicitly, we label the modified spans in The to make the rewriting more control- lable. Specifically, we add special tokens to Use to indicate the insertion, substitution, and deletion operations needed to convert x to Urooe
fine-tuned on
5 Experimental Setup
5.1 Datasets
Experiments are carried out on two widely used multi- molti-domain TOD datasets, MultiWOZ 2.1 (Eric et al., 2021) and Schema-Guider Dialog dataset (SGD) (Rastogi et al., 2020). We use the datasets processed by ConvLab-3 (Zhu al., 2022). The SGD dataset covers 45 services from 20 domains. some of which may have multiple services with dif- ferent schemas. We remove the Messaging domain since there are no single-domain dialogs this do-
Then the model used to rewrite the user ut- terance U in the second (and subsequent) single- domain dialog to Ucone when L, contains target

---

Domains (Comb.)
Single-domain Dialogs
Multi-domain Dialogs (CD)
Model
Epoch
Batch
Initialization
Training set
Dataset
QADST
512
UnifiedQA-v2
SINGLE
MultiWOZ
5 (14)
2824
7018 (29.8%)
CoQR-R
512
T5-Large
CANARD
SGD
19 (42)
2456
3387 (73.8%)
Value Tagger
128
RoBERTa-Large
TaskMaster
Table 4: Statistics of the MultiWOZ and SGD datascts.
CoQR Zero
512
T5-Large CoQR-Zero
CANARD
CoQR-SYN
512
SYN
main. Unlike Rastogi et al. (2020), assume the services of a dialog are not known beforehand. Therefore, we only retain one randomly chosen ser- vice for each domain to avoid confusion. The data statistics are shown in Table 4.
DomCLS
128
T5-Large
DST
128
T5-Large
Table 5: Training settings.
SINGLE: Single-domain dialogs.
The training. validation, and test sets of the origi- nal dataset merged and randomly re-partitioned minimize the data distribution gap between train- ing and testing. Single-domain dialogs are divided into training. validation, and test sets in a ratio of 8:1:1. 10% and 50% of the multi-domain dialogs are used for validation and testing, respectively, while the remaining 40% are reserved for experi- ments that require access to multi-domain dialogs. Domain combinations that have less than 10 di- alogs are ignored.
SINGLE+CONCHO Randomly select and con- catenate two single-domain dialogs of different domains (CONCAT). Similar to our method, the ending turn of the first dialog is removed. Train- ing models on SINGLE and CONCAT jointly.
SINGLE+SYN: Training models on SINGLE and our synthesized multi-domain dialogs jointly.
Unless otherwise specified, CONCAT and SYN have the same number of dialogs as SINGLE.
We also explore generating self-contained rewrites Use for all user utterances Uoone in a di- alog before DST during testing. The DST model is trained on SINGLE+ CONCAT data that do not contain cross-domain value transfer. We consider the following CoQR models to generate rewrite U.c given the context C2 and user utterance Uponw;
5.2 Metrics
The most widely used metric for DST is Joint Goal Accuracy (JGA), which evaluates whether the pre- diction BY and gold state B₃ are exactly matched. However, this metric is not smooth. If the model mispredicts even single turn, this metric will ig- nore predictions made in all other turns. Therefore, we use Relative Slot Accuracy (RSA)(Kim)
CoQR-Zero: A CoQR model trained CA- NARD dataset (Elgohary et 2019).
al., 2022), which can be regarded as soft version of
CoQR-SYN: Fine-tune CoQR-Zero on our SYN
dialogs that contain original single-domain utter- ances (U,9)) and rewritten utterances (Urone)) CoQR-ChatGPT: Prompting ChatGPT to gen- erate self-contained rewrites (see Appendix C).
JGA and can better distinguish models. It is calcu-
lated as [ where [B O B is the number of correctly predicted slots and |S| is the number of unique slots in B, U B,, We ignore the turns where |S| = 0, which means both B, and Bj, are empty. metric CDTA (Cross-domain Turn Accuracy) to measure a model's ability to capture the cross-domain value transfer behavior. CDTA is the Turn Accuracy (TA) cross-domain turns, examining turn prediction T4 that updates B to B, Following Dey et al. (2022), instead of comparing L1 and L,, we regard a turn t as correct when L3 C B, and L, B/. which credit models for correcting error predictions in previous turns.
The training details of mod- els are summarized in Table 5. Some models are trained for one epoch to avoid overfitting. Dom- CLS is trained on the same data as DST models. We train the value tagger using Adam optimizer with a linear learning rate schedule that initiates at 2c-5, For other models, we use Adafactor optimizer with a linear learning rate schedule that initiates at le-3. To generate diverse rewrites Cromv for our data synthesis, we random sampling for the CoQR-R model. For CoQR-Zero and CoQR-SYN that generate self-contained rewrites U we use the beam search algorithm with a beam size of 5. For QADST and DST models, we use the greedy decoding strategy.
Hyper-parameters
We propose
5.3 Baselines and Training Details
To verify the effectiveness of our proposed data synthesis framework, we compare DST models trained on different data:

---

SGD domains)
MuldWOZ (5 domains)
RSA
RSA
CDTA
CDTA
Training Data
DAC
4
94.1
44.9
23.5
13.2
2.7
85.9
44.4
22.1
1.6
SINGLE
93.8
71.8
59.5
48.4
11.8
85.1
65.2
54.1
TS-DST
X
94.0
78.1
53.5
32.9
1.6
85.3
83.9
44.9
11.2
SINCLE+CONCAT
/
93.9
76.9
74.7
50.9
1.8
85.3
83.6
68.4
11.1
94.0
81.8
54.3
33.1
25.8
85.1
82,4
50.7
20.9
SINGLE+SYX
92.9
81.3
77.0
66.7
26.9
84.8
81.8
70.4
30.8
92.3
47.4
12.9
83.0
52.7
37.2
SINGLE
93.5
64.3
56.2
37.0
83.9
59.3
44.4
MinTL
92.8
70.0
67.8
54.3
1.1
84.2
81.5
64.0
5.8
CINGLE+CONCAT
V
93.4
74.4
72.2
58.3
0.5
84.4
82.0
65.1
7.3
X
91.7
70.2
64.0
54.5
18.2
83.5
80.0
66.3
27.5
SINGLE+SYN
V
92.9
78.0
73.4
62.9
21.8
83.6
80.4
66.3
29.1
X 93.6
59.0
44.7
47.7
3.8
86.2
58.0
39.8
1.4
SINGLE
/
94.7
74.7
66.2
54.6
13.2
86.2
66.9
54.0
8.5
SDP-DST
81.9
66.6
87.2
84.9
68.5
SINGLE+CONCAT
78.3
77.5
58.7
1.7
84.3
68.1
X
92.6
83.4
80.1
74.6
18.1
86.9
83.8
72.4
26.8
SYN
94.8
81.9
78.4
66.4
19.7
86.2
82.6
70.6
27.7
Table 6: DST performance on real multi-domain dialogs. We divide turns based on the number of domains B3g and evaluate RSA separately. When there are more domains in in test dialogs than training dialogs, the better result between models with/without DAC paradigm is in gray The best CDTA for each model is in bold.
SGD
MultWWZ
tion test dialogs involving more domains than training dialogs. SDP-DST already generalizes well when trained on synthesized multi nulti-domain dialogs, and DAC leads to worse generalization. possible reason is SDP-DST-DAC have not seen samples consisting of dialog and slots inactive domains during training, thus may be confused by similar slots when predicted active domains are wrong. However, SDP-DST is in- feasible when there numerous domains due low efficiency, and the DAC paradigm is rea- sonable solution that substantially improves the training and inference efficiency.
Training Data
Single
Multi
Single
Multi
SINGLE
98.7
85.3
99.8
82.8
HIGGLECCONCAT
98.6
90.7
99.8
94.8
SINGLE+SYN
98.5
90.9
99.7
94.4
Table 7: Domain classifier accuracy.
6 Experiments and Analysis
6.1 Main Experiment
We first examine the performance of the domain classifier, which used by all models based on the DAC paradigm. As shown in Table 7, Dom- CLS accurately classifies single-domain dialogs and performs well on real multi-domain dialogs when trained on synthesized dialogs, allowing us track the state of each domain separately.
Compared with CONCAT, training models on our synthesized data SYN results in significantly bet- ter CDTA, which indicates SYN effectively en- ables models to capture value transfer between domains. Combining DAC and SYN leads to the best CDTA.
Then we conduct experiments on three DST models to verify the effectiveness using the DAC paradigm and our synthesized dialogs. Table shows model performance on real multi-domain dialogs. We have the following findings:
6.2 Generalization Unseen Domain Combinations
Models trained only on single-domain dialogs generalize poorly dialogs involving more than one domain. Using the DAC paradigm or syn- thesizing multi-domain dialogs for training can significantly improve model generalization.
We further investigate model generalization seen domain combinations. We first sample 15 do- mains from all 19 domains of the SGD dataset and then sample 10 domains from these 15 domains, creating D.15 and D.10, respectively. Then we train the model on the same amount of synthesized multi-
For T5-DST and MinTL, applying the DAC paradigm often leads to better model generaliza-

---

T5-DST
RSA
CDTA
SGD
MultWWZZ
T5-DST-DAC
RSA
CDTA
RSA
CDTA
DAC Domains Dau D10:1
Dis.
Dao
D10.15 D10.19
SINGLE*CONCAT
82.5
1.8
83.2
11.1
D.w
83.4
80.4
74.7
30.6
17.0
14.5
w/ CoQR-Zero
75.8
16.6
78.9
11.4
x Dan
83.1
85.1
76.3
44.2
23.9
19.9
w/ CoQRRChatGPT
80.5
26.8
81.4
24.7
Das
81.3
84.8
81.6
33.5
27.2
23.6
w/ COQRR-YNN
84.1
17.2
83.1
24.0
D.10
87.0
84.1
82.1
38.0
27.7
21.0
SINGLE+SYI
84.8
26.9
82.2
30.8
/
D.15
85.3
85,5
83.0
32.2
27.4
20.4
w/o utterance rewrite
82.8
4.4
82.7
20.2
D.1
87.5
85.6
83.7
38.4
27.2
24.3
w/o filter
84.4
21.1
81.5
27.6
0.5x SYN data
85.0
24.0
81.7
28.4
Table 8: Model generalization on unseen domain combi- nations. The test multi-domain dialogs divided into three disjoint sets based on their active domains: all in D. least one 15 and at least one in 19- Results for dialogs with unseen domain combinations are in gray
2x SYN data
83.8
26.5
81.2
29.8
doms-d slot pairs
F1 >> 0
84.6
26.2
81.9
29.1
F1 > 0.1
84.8
26.9
82.2
30.8
F1 0.3
85,2
24.0
81.8
30.6
P1 0.5
85.1
26.2
81.5
26.9
F1 0.8
83,9
18.1
81.7
21.5
domain dialogs as SYN but involving domains in Da1//:11 only, and evaluate model performance on dialogs containing unseen domain combinations (at least one domain in D10:19/D15:19). As shown in Table 8, the DAC paradigm makes T5-DST gen- eralize better unseen domain combinations (gray cells). Unlike T5-DST, synthesizing part of domain combinations sufficient for T5-DST-DAC per- form well on all domain combinations.
Mined from MULTI
86.4
30.6
82.7
34.7
90% S0 86.7
39.6
83.4
31.4
Table
CoQR on on-the-fly fly experiments and ablation study. Results of using original SYN data are in gray
Performance does not change much when the Fl threshold of filtering slot pairs mined by QADST varies from 0.3. Higher F1 will filter out real related slots and lead worse CDTA. The results of using slot pairs mined from real multi-domain dialogs for data synthesis suggest that improving slot relation mining can further enhance CDTA. Compared with replacing 10% single-domain di- alogs with real multi-domain dialogs, using SYN for training achieves comparable results Mul- tiWOZ but worse results SGD which contains much more domains. This indicates that the gap between real data and SYN is large on SGD, call- ing for better data synthesis methods.
6.3
CoQR On-the-Fly
The synthesized multi-domain dialogs can also be used to train CoQR model to rewrite the utterance to express values explicitly, resolving cross-domain value transfer before DST. We compare the CoQR model trained on SYN with other CoQR models in Table 9. The DST model is T5-DST-DAC trained on CONCAL We find that rewriting user utterances with CoQR-Zero or ChatGPT before DST severely sacrifices the overall performance for improving CDTA. In contrast, using CoQR-SYN trained on our synthesized rewrites can effectively address cross-domain value transfer while main- taining the overall performance. This is because CoQR-SYN is trained in-domain data and there- fore knows when and how rewrite. We observe similar results on other DST models (Appendix A).
7 Conclusion
In this paper, we investigate how to build multi- domain DST models using only single-domain di- alogs. We propose divide-and-congee paradigm and multi-domain dialog synthesis framework, en- abling DST models generalize better on unseen multi-domain dialogs and capture value transfer between domains. The DAC paradigm simplifies multi-domain DST to single-domain DST, improv- ing model generalization and efficiency. Our syn- thesized multi-domain dialogs can be used to train DST models directly or train CoQR model to rewrite utterances before DST. Both usages can sig- nificantly improve the domain relation modeling ability of DST models. Our methods can boost the development of multi domti- DST models by re- ducing the cost of collecting multi-domaii dialogs.
6.4 A Ablation Study and Robustness Analysis We conduct comprehensive experiments to identify key factors in data synthesis. Results are shown in Table 9. We have the following findings:
Without utterance rewriting or value filter, the CDTA decreases significantly, indicating that high-quality rewrites teach models to capture the cross-domain value transfer behavior.
Using 50% SyN data decrease the CDTA little, while using twice SYN data does not guarantee performance improvement.

---

8 Limitations
oriented dialogue modelling, In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
While our proposed methods have made initial progress in building multi-domain DST from singlc-domain dialogs, there is still much room for improvement. Currently, the DAC paradigm has two main drawbacks: (1) models could not ac- cess the dialog history of other domains and thus can only transfer values presented in the state, (2) the domain elassifier and DST model are optimized independently, which may cause error propagation. Future works can retrieve relevant context dynam- ically and incorporate DST feedback in domain classification. The scope of multi-domain dialogs considered in this paper is rather simplistic, missing many kinds of domain interactions, such domain composition (Andreas al., 2020). The gap be- tween synthesized multi-domain dialogs and real ones is still large, which can be improved through (1) utilizing commonsense knowledge in slot rela- tion mining, (2) modeling diverse discourse struc- tures, and (3) generating more reasonable rewrites with stronger CoQR models.
Bill Byrne, Karthik Krishnamoorthi, Saravanan Ganesh, and Mihir Kale. 2021. TicketTalk: Toward human- level performance with end-to-end, transaction- dialog systems. In Proceedingg of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joins Conference Natural Language Processing (Volume 1: Long Papers), pages 67 1-680, Online. Association for Computational Linguistics.
Bill Byme, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz. Amit Dubey, Kyu- Young Kim, and Andy Cedilnik. 2019. Taskmaster-1: To- ward a realistic and diverse dialog dataset. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLPP) pages 4516-4525, Hong Kong, China, Association for Computational Linguistics.
Lu Chen, Boer Lv, Chi Wang. Su Zhu, Bowen Tan, and Kai Yu. 2020. Schema-guides multi-domain dialogue state tracking with graph attention neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 7521-7528.
A Acknowledgements
This work was supported by the National Key Re- search and Development Program of China (No. 20212D01 This work was also supported by the National Science Foundation for Distin- guished Young Scholars (with No. 62125604) and the NSFC projects (Key project with No. 61936010).
Suvodip Dey, Ramamohan Kummara, and Maunendra Desarkar. 2022. Towards fair evaluation dialogue state tracking by flexible incorporation of turn-level performances. In Proceedings the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers). pages Dublin, Ireland. Association for Computational Lin- guistics.
Ahmed Elgohary. Denis Peskov, and Jordan Boyd- Graber. 2019. Can you unpack that? learning to rewrite questions-in-comtext. In Proceedings of the 2019 Conference Empirical Methods in Natu- rul Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- pages 5918 5924, Hong Kong. China. Association for Computational Linguistics.
References
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dornrr, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho. Wendy Iwaszuk. Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy Mc- Govern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, An- drei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020. Task-oriented dialogue as dataflow synthesis. Trans- actions of the Association for Computational Linguis- tics, 8:556-571.
Mihail Eric, Nicole Chartier, Behnam Hedayatnia, Karthik Gopalakrishnan, Pankaj Rajan, Yang Liu, and Dilek Hakkami-Tur. 2021. Multi-sentence knowl- edge selection in open-domain dialogue. In Proceed- ings of the 14th International Conference on Natural Language Generation, pages 76-86, Aberdeen, Scot- land, UK. Association for Computational Linguistics.
Yue Feng, Aldo Lipani, Fanghua Ye, Qiang Zhang, and Emine Yilmaz. 2022. Dynamic schema graph fusion network for multi-domaie dialogue state tracking. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 115-126. Dublin, Ireland. Association for Computationa Linguistics.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašié. 2018. MultiWOZ- a large- scale multi-domain Wizard-of-OO Oz. dataset for task-

---

Michael Heck, Carel van Nickerk, Nurul Lubis, Chris- tian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gasic. 2020. TripPy: A triple copy strategy for value independent neural dialog state tracking. In Proceedings of the 21th Annual Meeting of the Special Interesi Group Discourse and Dialogue, pages 35-44, 1st virtual meeting. Association for Computational Linguistics.
Shiyang Li, Semih Yavuz, Kazuma Hashimoto, Jia Li. Tong Niu, Nazneen Rajani, Xifeng Yan, Yingbo Zhou, and Caiming Xiong. 2021. Coco: Control- lable counterfactuals for evaluating dialogue state trackers. In International Conference on Learning Representations.
Weizhe Lin, Bo-Hsiang Tseng, and Bill Bymee 2021a. Knowledge-awand graph-enhanced GPT-2 for dia- logue state tracking. Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7871-7881, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In Ad- vances in Neural Information Processing Systems, volume 33, pages 20179-20191. Curran Associates, Inc.
Zhaojjang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon. Zhenpeng Zhou, Paul Crook. Zhiguang Wang. Zhou Yu, Eunjoon Cho, Rajen Subba, and Pascale Fung. 2021b. Zero-shot dialogue state tracking via cross-task transfer. In Proceedings of the 202/ Con- Serence on Empirical Methods in Natural Language Procexsing, pages 7890-7900, Online and Punta Cana, Dominican Republic. Association for Com- putational Linguistics.
Fangkai Jiao, Yangyang Guo, Minlie Huang, and Liqiang Nie. 2022. Enhanced multi-domain dia- logue state tracker with second-order slot interac- tions. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:265-276.
Daniel Khashabi. Yeganeh Kordi, and Hannaneh Ha- jishirzi. 2022. Unifiedqa-v2: Stronger generalization via broader cross-format training. arXiv preprint 222i::2902.12359.
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. 2020. MinTI: Minimalist trans- fer learning for task orientere dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3391-3405, Online. Association for Computa- tional Linguistics.
Sungdong Kim, Minsuk Chang, and Sang-Woo Lee. 2021. WeuralWOZ: Learning to collect task-oriented dialogue via model-based simulation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1tth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3704-3714 Online. Association for Computational Linguistics.
Biswesh Mohapatra, Gaurav Pandey, Danish Contrac- tor, and Sachindra Joshi. 2021. Simulated chats for building dialog systems: Learning to generate conver- sations from instructions. In Findings of the Associ- ation for Computational Linguistics: FMNLP 2021, pages 190-1203, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sang- Woo Lee. 2020. Efficient dialogue state tracking by selectively overwriting memory. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 567-582, Online. Association for Computational Linguistics.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan- deh, Lars Liden, and Jianfeng Gao. 2021. Soloist: Building task bots scale with transfer learning and machine teaching. Transactions of the Association for Computational Linguistics, 9:807-824.
Takyoung Kim, Hoonsang Yoon, Yukyung Lee, Pilsung Kang, and Misuk Kim. 2022. Mismatch between multi-turn dialogue and its evaluation metric in dia- logue state tracking. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 297- 309, Dublin, Ireland. Association for Computational Linguistics.
Jun Quan, Deyi Xiong, Bonnie Webber, and Changjian Hu. 2019. GECOR: An end-to-end generative el- lipsis and co-reference resolution model for task- oriented dialogue. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4547-4557, Hong Kong, China, Association for Computational Linguistics.
thia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021. Dialogue state tracking with a language model using schema-driven prompting. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4937-4949, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Hwaran Lec, Jinsik Lee, and The-Yoon& Kim. 2019. SUMBT: Slot-utterance matching for universal and scalable belief tracking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5478-5483, Florence, Italy. Asso- ciation for Computational Linguistics.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards

---

scalable multi-domain conversational agents: The schema-guide dialogue dataset. In Proceedings of the AAAl Conference Artificial Intelligence.
Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT pre training approach with post-training. In Proceedings of the 20th Chinese National Conference Computational Linguisties, pages 1218-1227, Huhhot, China. Chinese Informa- tion Processing Society of China.
Pushpendre Rastogi, Arpit Gupta, Tongfei Chen, and Mathias Lambert. 2019. Scaling multi-domain dia- logue state tracking via query reformulation. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 2 (Industry Papers). pages 97-105, Minneapolis, Minnesota. Association for Computational Linguis- tics.
Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Peng- wei Hu. Cheng Niu, and Jie Zhou. 2019, Improv- ing multi-turn dialogue modelling with utterance ReWrit.r. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 22-31, Florence, Italy. Association for Com- putational Linguistics.
Dazhen Wan, Zheng Zhang. Qi Zhu, Lizi Liao, and Min- lie Huang. 2022. unified dialogue user simulator for fho-shot data augmentation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3788-3799, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini- Asl. Caiming Xiong. Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Compu- lational Linguistics, pages 808-819, Florence, Italy. Association for Computational Linguistics.
Fanghua Ye, Jarana Manotumruksa, Qiang Zhang, Shenghui Li, and Emine Yilmaz. 2021. Slot self- attentive dialogue state tracking. In Proceedings of the Web Conference 2021, pages 1608-1608.
Jinguu Zhang, Kazuma Hashimoto, Chien-Shaer Shenn Wu. Yao Wang, Philip Yu, Richard Socher, and Caiming Xiong. 2020. Find classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. In Proceedings of the Ninth Joint Confer- ence on Lexical and Computational Semantics, pages 154-167, Barcelona, Spain (Online). Association for Computational Linguistics.
Qi Zhu, Christian Geishauser, Hsien chin Lin, Carel Niekerk, Baolin Peng, Zheng Zhang, Michael Heck, Nurul Lubis, Dazhen Wan, Xiaochen Zhu, Jianfeng Gao, Milica Gašić, and Minlie Huang. 2022. Convlab-3: A flexible dialogue system toolkit based on a unified data format. arXiv preprint arXiv:2211.17148.
Su Zhu, Jieyu Li, Lu Chen, and Kai Yu. 2020. Effi- cient context and schema fusion networks for multi- domain dialogue state tracking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 766-781, Online. Association for Com- putational Linguistics.

---

A Full Results of CoQR On-the-Fly
We show the full results of CoQR on-the-fly experi- ments in Table 10. From the results, we can observe that using the CoQR model to rewrite utterances be- fore DST can improve CDTA. However, the CoQR model trained CANARD (CoQR-Zero) severely decreases the overall performance (RSA). Using ChatGPT (gpt-3. 1-1--33.-tuuubr 5-turbo-0301 0301) rewrite utter- ances reduces the performance drop and achieves the best CDTA among CoQR models. We prompt ChatGPT with dialog history and ask it to rewrite the user utterance to resolve any anaphora or ellip- sis. Several examples we used to prompt ChatGPT are demonstrated in Table 12. Using the CoQR model trained on SYN data (CoQR-SYN) can im- prove CDTA and maintain RSA. Directly training DST models on SYN data is better than using SYN data to train CoQR model to rewrite utterances.
Full Results of Slot Relation Mining
To evaluate the synthesized slot pairs, extract "golden" slot pairs from real multi-domain dialogs, which may have noise. According the state notation, we extract the (source, target) slot pairs Vtarget.alue==suureee value ssurce. is not empty) larger than 10%.
We further report some statistics about the slot pairs in SYN data in Table 11. Increasing the Fl threshold will decrease the recall of golden slot pairs but improve precision. However, recalling more golden slot pairs seems more important than inereasing precision. Comparing SGD and Mul- tiwOZ, we find that our data synthesize frame- work can recall relatively more golden slot pairs on MultiWOZ, which may explain the performance gap between using SYN data and using real multi- domain data for training in Table 9. On SGD, even if we use the golden slot pairs for data synthesis, SYN data not cover all golden slot pairs (only 75%) due the value filtering process. On the contrary, we can almost synthesize all golden slot pairs (97.1%) on MultWWOZ.
Prompt for ChatGPT to Generate Self-contained Rewrites
Table 12 shows the prompt we used for ChatGPT to rewrite the user utterance in the CoQR on-the-fly experiments. All user utterances in the dialog will rewritten to resolve cross-domain value transfer.

---

SGD
MultWWZZ
RSA
CDTA
RSA
CDTA
LINGLE+CONCAL
82.5
1.8
83.2
11.1
w/ -Zro
w/ CoQR-
75.8
16.6
78.9
11.4
T5-DST-DAC
80.5
26.8
81.4
24.7
w/ CoQR-SYN
84.1
17.2
83.1
24.0
SINGLE+SYN
84.8
26.9
82.2
30.8
SINGLE+CONCAT
80.9
0.5
81.8
7.3
w/ CoQR- -ero
73.3
16.0
77.5
8.1
MinTL-DAC
w/ CoQR ChatGPT
78.4
25.7
80.0
23.0
w/ CoQR-SYS
82.7
16.9
81.5
21.6
SINGLE+SYN
82.7
21.8
80.7
29.1
SINGLE+ SINGLE+CONCAT CONCAT
84.0
1.7
84.1
8.3
-eroo w/ CoQR- Zero w/ COQR-ChatGPT
76.5
17.9
79.6
9.4
SDP-DST-DAC
81.2
27.3
82.3
24.8
w/ CoQR-SYN
85.6
17.8
83.8
22.6
SINGLE+SYN
86.0
19.7
83.2
27.7
Table 10: CoQR on-the- fly experiments. Results of using original SYN data are in gray
15-DST-DAC
SGD
MultiWOZ
Syn Slot Pairs
RSA
CDТA
Ratio
Precision
Recall
RSA
CDTA
Ratio
Precision
Recall
F1 F1
84.6
26.2
10.9
58.6
81.9
14.9
29.1
88.2
26.9
82.2
30.8
11.2
79.4
F1 0.3
85.2
4.1
12.0
56.3
81.8
30.6
8.7
41.7
73.5
F1
26.2
11.4
49.4
81.5
26.9
7.1
46.9
67.6
> 0.8
83.9
18.1
2.8
14.9
47.1
81.7
21.5
2.5
64.7
32.4
Mined from MULTI
86.4
30.6
0.7
100.0
75.9
82.7
34.7
4.8
100.0
97.1
Table 11: Training on SYN data constructed using different slot pairs. "Ratio" is the ratio of remaining slot pairs SYN data compared all possible slot pairs. "Precision" and "Recall" is calculated by comparing slot pairs in SYN data and slot pairs mined from MULTI. Significant performance drops are in gray
Replace the anaphora or ellipsis in this sentence with its real value in the context. Example 1: Edit a sentence "I'd ulso like to find a restaurant there". The context is ["I'm bored. Can you find me a movie? I'm at Newark.", "There's s 9 movies playing, including Breakthrough, Captain Marvel, and Dumbo."] "J. "there" pronoun refering a place, and it refers to "Newark" according to the context. So the edited sentence should be "I'd also like to find restaurant at Newark".
Example 2:
Edit a sentence "no, but it should have free parking, please.". The context is ('Hi. I'm looking for hotel in the east. The internet is not needed.", "There are no hotels that do not have internet, but 7 which do have it. Do you have specific price range you'd 'dd prefer?"]. "it" refers the hotel. So the edited sentence should be "no, but the hotel should have free parking. please.".
Example 3:
Edit sentence "yeah book it for the same group of people please". The context is ["I'm looking train that is leaving Thursday but will arrive by 09:30 to Cambridge." 'Okay. and where you leaving from?', "i will depart from birmingham street', 'TR3736 arrives in Cambridge 09:23.', "That would be great. Please book tickets for 7 people. I'll also need reference number, if possible. also need place stay in the north. I'd like something that includes free parking.". "Okay and what is your price range"', "It doesn't 't really matter long the parking free. I don't Thursday. "Ok great. highly suggest the Acorn Guest House. Would you like for to book room?']. The same group people refers 7 people the context. So the edited sentence should be "yeah book for the same group of people please".
'Okay booked it for you and your reference number is OLYLOB3V", "Thank you. will
need internet." 'when your stay?". "Let's go with
If anaphora ellipsis exists, the edited utterance remains unchaneged. The sentence you are supposed edit "[utterance]" and context (history). edited sentence should be:
Table 12: The text input we used to prompt ChatGPT to rewrite the user utterance to resolve anaphora and ellipsis.


Reviews:
[{"review_id": "uCHUYIZZaE", "reviewer": "Reviewer_NSN3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and feedback. We appreciate that you found our methods are effective and the improvements are significant. We have carefully considered your comments and would like to respond point-by-point:\n\n> Error propagation could be an issue as the domain classifier and state tracker are optimized independently in the divide-and-conquer paradigm.\n\nWe agree this is a limitation of the current divide-and-conquer (DAC) paradigm. One of the reasons that we choose to optimize the domain classifier and DST models independently is to make a fair comparison between different DST models with the same domain classifier. Future works can improve the DAC paradigm by correcting the domain classifier's prediction with feedback from the DST model.\n\n> The association between DAC method and multi-domain dialog synthesis framework is relatively weak, insufficient to constitute a cohesive unified framework.\n\nWe consider building multi-domain DST using single-domain data in two perspectives: model and data. Investigating only one of them could not show its necessity since one could argue that the problem can be solved by improving the other (better model/data synthesis method). Indeed, our experiments show that both the DAC paradigm and the multi-domain data synthesis framework are necessary and complement each other.\n\n> Wrong template is used: \"Anonymous ACL submission\"\n\nThanks for your kind reminder! We will modify this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes two novel methods, including divide-and-conquer paradigm and data synthesis framework, to address the key challenges of building multi-domain models without multi-domain data.\n2. Comprehensive experiments demonstrate the effectiveness of the proposed methods on two benchmark datasets using three representative dialog state tracking models. Results show significant improvements in model generalization and capturing cross-domain value transfer.", "reasons_to_reject": "1. Error propagation could be an issue as the domain classifier and state tracker are optimized independently in the divide-and-conquer paradigm.\n2. The association between DAC method and multi-domain dialog synthesis framework is relatively weak, insufficient to constitute a cohesive unified framework.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on improving multi-domain DST with single-domain data. It proposes a new divide-and-conquer approach and data synthesis method. The method reduces the need for expensive multi-domain dialogues labeling. The results demonstrate the cross-domain generalization.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Wrong template is used: \"Anonymous ACL submission\""}, "extra_text": {}}, {"review_id": "jBpcKXUmgE", "reviewer": "Reviewer_DJWY", "summary": "", "questions": "- In section 5.2.1, the authors said “For a dialog of domain d, the model is trained to predict the values of slots in d. When performing cross-domain inference, the model predicts the values of target slots in other domains.” This is a bit unclear for me. How to get all candidate slots and values in cross-domain inference? Do you iterate over all of the domains except d to get all candidate slots, or train a dedicated model for each domain?\n- Theoretically, the information of belief states of upstream domains is enough if the correct belief states are sent to the downstream domains. However, the errors in the belief states may also propagate to downstream domains. How to avoid this in the divide-and-conquer DST paradigm?\n- How to make sure the slot pairs that could potentially take the same value in a specific dialog context really make sense? Is there any method to validate if the synthetic dialogs are reasonable?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and feedback. We appreciate that you found (1) the problem we investigate is interesting, and our solution is feasible and effective; (2) our proposed paradigm can be generalized to a series of DST models; (3) the paper is well-written and easy to follow.\n\nWe have carefully considered your comments and would like to respond point-by-point:\n\n> The divide-and-conquer DST paradigm is to model multi-domain DST in a pipelined single-domain manner with the dialog states from the upstream domains. The pipeline solutions will suffer from some intrinsic weaknesses such as error propagation. In contrast, previous multi-domain DST work can be seen as an end-to-end solution since they can always generate or update all domains and slots in one turn.\n\nWe agree this is a limitation of the current divide-and-conquer (DAC) paradigm. One of the reasons that we choose to optimize the domain classifier and DST models independently is to make a fair comparison between different DST models with the same domain classifier. Compared with DST models that always predict all domains and slots, models applying the DAC paradigm are more efficient and generalize better on multi-domain dialogs that have unseen domain combinations (Table 8) and more domains than training (Table 6). Therefore, although the DAC paradigm has some weaknesses, we think it is more suitable for the scenario where the distribution of real multi-domain dialogs is unknown.\n\n> Considering the data is synthetic rather than natural, it may introduce errors in the multi-domain dialog synthesis framework. Although the author proposes a filtering process by a value tagger, it cannot make sure that the automatically mined slot pairs indeed make sense.\n\nWe agree. Actually, we do observe some erroneous slot pairs. We think the slot mining process can be further improved by using large language models that possess commonsense knowledge. We do not use ChatGPT to do this because slot pair mining is the first step, and using ChatGPT (not open-source, not stable) will greatly affect the reproducibility of the work. At the time we conduct experiments, open-source LLMs like LLaMa are not available. In the future, we will use these open-source LLMs to mine slot pairs that could potentially transfer value.\n\n> In section 5.2.1, the authors said “For a dialog of domain d, the model is trained to predict the values of slots in d. When performing cross-domain inference, the model predicts the values of target slots in other domains.” This is a bit unclear for me. How to get all candidate slots and values in cross-domain inference? Do you iterate over all of the domains except d to get all candidate slots, or train a dedicated model for each domain?\n\nWe train a single model on all single-domain dialogs. For each single-domain dialog belonging to domain $d$, the model is only optimized to predict the slots of $d$. During cross-domain inference, for a single-domain dialog, we iterate over all of the domains except $d$ to get all candidate slots.\n\n> Theoretically, the information of belief states of upstream domains is enough if the correct belief states are sent to the downstream domains. However, the errors in the belief states may also propagate to downstream domains. How to avoid this in the divide-and-conquer DST paradigm?\n\nThis question enlightens us! Actually, this could be one of the reasons that SDP-DST (predicts all slots at each turn with all dialog history as input) performs better than SDP-DST with DAC (predicts slots of the active domain at each turn with only current domain dialog as input) in Table 6. To alleviate this problem, the DAC paradigm may also take relevant dialog context as input through retrieval. For example, when synthesizing the turn $t$ that has target slots for cross-domain value transfer, we can also create retrieval samples for retrieving turns $(t^\\prime-w, t^\\prime+w)$, where $t^\\prime$ is the turn that source slots appear, and $w$ is the context window size. In this way, the DST models can directly refer to the relevant context instead of the model-predicted state when solving cross-domain value transfer.\n\n> How to make sure the slot pairs that could potentially take the same value in a specific dialog context really make sense? Is there any method to validate if the synthetic dialogs are reasonable?\n\nAs mentioned above, we plan to use LLMs that possess commonsense knowledge to identify slot pairs that could potentially take the same value in a specific dialog context. We think it is also a plausible way to validate if the synthetic dialogs are reasonable.\n\nThank you for your enlightening comments! We will continue refining our work according to your advice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The question raised in this paper is interesting and valuable in real-world scenarios: How to train an effective multi-domain DST model when we only have single-domain dialogs?\n- This paper proposes a feasible solution to tackle this problem. The paper is well-written and easy to follow.\n- The paradigm proposed in this paper can be generalized to a series of DST models.\n- The experimental results demonstrate the effectiveness of the proposed solution.", "reasons_to_reject": "- The divide-and-conquer DST paradigm is to model multi-domain DST in a pipelined single-domain manner with the dialog states from the upstream domains. The pipeline solutions will suffer from some intrinsic weaknesses such as error propagation. In contrast, previous multi-domain DST work can be seen as an end-to-end solution since they can always generate or update all domains and slots in one turn.\n- Considering the data is synthetic rather than natural, it may introduce errors in the multi-domain dialog synthesis framework. Although the author proposes a filtering process by a value tagger, it cannot make sure that the automatically mined slot pairs indeed make sense.", "questions_to_authors": "- In section 5.2.1, the authors said “For a dialog of domain d, the model is trained to predict the values of slots in d. When performing cross-domain inference, the model predicts the values of target slots in other domains.” This is a bit unclear for me. How to get all candidate slots and values in cross-domain inference? Do you iterate over all of the domains except d to get all candidate slots, or train a dedicated model for each domain?\n- Theoretically, the information of belief states of upstream domains is enough if the correct belief states are sent to the downstream domains. However, the errors in the belief states may also propagate to downstream domains. How to avoid this in the divide-and-conquer DST paradigm?\n- How to make sure the slot pairs that could potentially take the same value in a specific dialog context really make sense? Is there any method to validate if the synthetic dialogs are reasonable?", "paper_topic_and_main_contributions": "This paper discusses building multi-domain Dialog State Tracking (DST) models using single-domain dialogs. The main problem the paper addresses is the difficulty and expenses involved in manual domain relation definition and multi-domain dialog data collection. Hence, the authors proposed a divide-and-conquer (DAC) DST paradigm and a multi-domain dialog synthesis framework. The divided and conquer approach generalizes better on dialogs involving unseen domain combinations while the multi-domain dialog synthesis framework simulates domain relationships by merging single-domain dialogs and modifying them. The experiments conducted on two datasets demonstrate the efficacy of these proposed strategies.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UkyLcLmdnc", "reviewer": "Reviewer_9qPF", "summary": "", "questions": "See above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and feedback. We appreciate that you found (1) our method is effective and can reduce the cost of multi-domain data collection; (2) our method is generic and potentially can be used in other scenarios relating to compositional generalization; (3) the writing is clear and easy to follow.\n\nWe have carefully considered your comments and would like to respond point-by-point:\n> 1. The author should provide discussions to support their motivation to work on DST, a component that serves task-oriented dialog systems (TOD). In particular, the authors should compare TOD with general-purpose chatbots like ChatGPT.\n\nOur research interest is task-oriented dialog (TOD), and we think the most challenging and valuable problem of multi-domain dialogs compared with single-domain dialogs is the recognition of cross-domain value transfer when domain relations are undefined, so we investigate multi-domain DST that needs to model value transfer between domains. We are unsure about the meaning of \"compare TOD with general-purpose chatbots like ChatGPT\":\n1. Did you mean using ChatGPT for TOD? Recent studies[1,2,3] show that ChatGPT achieves SOTA zero-shot DST results on TOD tasks but still largely underperforms fine-tuned models. So in this paper, we did not use ChatGPT for DST. \n2. Or did you mean we should discuss the difference between TOD and general-purpose chatbots? We do have a few opinions about the difference and what LLMs like ChatGPT need so as to serve as TOD systems. However, we do not include these ideas since they are not strongly related to the focus of this paper. \n\nWe would love to discuss with you to figure out what you think exactly.\n\n> 2. This paper only investigates generative DST models with only a fixed scale T5 as the backbone. Whether the improvement in compositional generalization remains considerable with non-generative methods and larger models remains unclear.\n\nThanks for your suggestions! As we stated in Section 3.2, we use open-vocabulary generative DST models because they are flexible to generate any value and require minimal annotation. Classification-based models need full value lists and can not predict values out of the lists, while extraction-based models need value span annotations and can not predict values that are not presented in the dialog context. Besides, most Large Language Models (LLMs) are generative models. So we conducted experiments using generative DST considering community interest. Although we do not try non-generative methods, we explore three different kinds of generative models to show the intrinsic difficulty in modeling cross-domain value transfer (often through anaphora) without real multi-domain data. In fact, we think most current multi-domain DST models would get into trouble with this problem, and our Divide-and-Conquer paradigm and data synthesis framework would help.\n\nWe use T5-large because it's the largest model we can afford to conduct comprehensive experiments. For LLMs that show emergent abilities[4], we do not have sufficient computation resources to fine-tune them. As stated above, without fine-tuning, ChatGPT could not give satisfactory DST performance[1,2,3]. \n\nWe do not intend to deny the limitations of our work. We just want to explain the reasons for our choices in detail, in response to your valuable feedback. Thanks for your suggestions. We will verify the effectiveness of our method on non-generative methods and larger models in the future. We kindly ask you to reconsider our paper.\n\n---\n\n[1] ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity? (Heck et al., ACL 2023)\n\n[2] Are LLMs All You Need for Task-Oriented Dialogue? (Hudeček and Dušek, SIGDial 2023)\n\n[3] A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understanding (Pan et al., arxiv 2023)\n\n[4] Emergent Abilities of Large Language Models (Wei et al., TMLR 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The result is positive and also helpful for reducing the cost of data collection for multi-domain DST.\n2. The method is data-centric and thus generic for various models. Potentially, this method can also benefit other domains relating to compositional generalization.\n2. The writing is clear and easy to follow.", "reasons_to_reject": "1. The author should provide discussions to support their motivation to work on DST, a component that serves task-oriented dialog systems (TOD). In particular, the authors should compare TOD with general-purpose chatbots like ChatGPT.\n2. This paper only investigates generative DST models with only a fixed scale T5 as the backbone. Whether the improvement in compositional generalization remains considerable with non-generative methods and larger models remains unclear.", "questions_to_authors": "See above.", "paper_topic_and_main_contributions": "This paper proposes a divide-and-conquer paradigm for dialog state tracing (DST). The method divides dialogs according to domains and mixt them to synthesis multi-domain data, providing a data augmentation method to improve cross-domain DST. Experiment results show this method boosts multi-domain DST performance using single-domain data.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes constructing multi-domain Dialog State Tracking (DST) models using a divide-and-conquer (DAC) DST paradigm along with multi-domain dialogue synthesis from existing single-domain dialogues.

Contributions:
 - The paper proposes two novel methods, including divide-and-conquer paradigm and data synthesis framework, to address the key challenges of building multi-domain models without multi-domain data.
 - The method is data-centric and thus generic for various models.
 - Experiments demonstrate the effectiveness of the proposed methods.
Decision: Accept-Main