Domain-slot Relationship Modeling using  a Pre-trained Language Encoder for Multi-Domain Dialogue State Tracking
Authors: Jinwon An
Misuk Kim
Sungzoon Cho
Junseong Bang
Venue:  (2021)
Keywords: Dialogue State Tracking
Domain-slot Relationship Modeling
Pre-trained Language Encoder

Abstract: Dialogue state tracking for multi-domain dialogues is challenging because the model should be able to track dialogue states across multiple domains and slots. Past studies had its limitations in that they did not factor in the relationship among different domain-slot pairs. Although recent approaches did support relationship modeling among the domain-slot pairs, they did not leverage a pre-trained language model, which has improved the performance of numerous natural language tasks, in the encoding process. Our approach fills the gap between these previous studies. We propose a model for multi-domain dialogue state tracking that effectively models the relationship among domain-slot pairs using a pre-trained language encoder. Inspired by the way the special $[CLS]$ token in BERT is used to aggregate the information of the whole sequence, we use multiple special tokens for each domain-slot pair that encodes information corresponding to its domain and slot. The special tokens are run together with the dialogue context through the pre-trained language encoder, which effectively models the relationship among different domain-slot pairs. Our experimental results show that our model achieves state-of-the-art performance on the MultiWOZ-2.1 and MultiWOZ-2.2 dataset.

Under review as a conference paper at ICLR 2021
DOMAIN-SLOT RELATIONSHIP MODELING USING A LANGUAGE ENCODER FOR MULTI- DOMAIN DIALOGUE STATE TRACKING
Anonymous authors Paper under double-blind review
ABSTRACT
Dialogue state tracking for multi-domain dialogues is challenging because the model should be able to track dialogue states across multiple domains and slots. Past studies had its limitations that they did not factor in the relationship among different domain-slot pairs. Although recent approaches did support relationship modeling among the domain-slot pairs, they did not leverage a pre-trained lan- guage model, which has improved the performance of numerous natural language tasks, in the encoding process. Our approach fills the gap between these previ- ous studies. We propose model for multi-domain dialogue state tracking that effectively models the relationship among domain-slot pairs using a pre-trained language encoder. Inspired by the way the special [CLS] token in BERT is used to aggregate the information of the whole sequence, we use multiple spe- cial tokens for each domain-slot pair that encodes information corresponding to its domain and slot. The special tokens are run together with the dialogue context through the pre-trained language encoder, which effectively models the relationship among different domain-slot pairs. Our experimental results show that our model achieves state-of-the-an performance on the MultiWOZ-2.1 and MultiWOZ- dataset.
INTRODUCTION
A task-oriented dialogue system is designed to help humans solve tasks by understanding their needs and providing relevant information accordingly. For example, such a system may assist its user with making reservation having a nice dinner. It can also recommend an attraction site to a travelling user, accommodating the user's specific preferences. Dialogue State Tracking (DST) oriented dialogue systems, which aims to identify the state of the dialogue between the user and the system. DST represents the dialogue state with triplets of the following items: domain, slot, a value. A set of {restaurant, price range, cheap), or of [train, arrive-by, 7:00 pmt are examples of such triplets. Fig. [ag.[iilinn Iillstrrates an example case of the dialogue state during the course of the conversation between the user and the system. Since a dialogue continues for multiple turns of utterances, the DST model should successfully predict the dialogue state at each turn as the conversation proceeds. For multi-domain conversations, the DST model should be able to track dialogue states across different domains and slots.
appropriate restaurant by understanding the user's needs for
core component of these task-
Past research on multi-domain conversations used placeholder in the model to represent domain- slot pairs. A domain-slot pair is inserted into the placeholder in each run, and the model runs repeatedly until it covers all types of the domain-slot pairs. (Wu et al. 2019; Zhang et al. 2019 Lee et al. 2019). A DST model generally uses an encoder to extract information from the dialogue con- text that is relevant to the dialogue state. A typical input for multi-domain DST model comprises a sequence of the user's 's and the system's 's utterances up to the turn t, X₂. and the domain-slot infor- mation for domain and slot j. D.Sj. In each run, the model feeds the input for given domain-slot pair through the encoder.
Sencader (X,, D,S,)) for 1,... ,, j 1,... 1....,90

---

Under review as a conference paper at ICLR 2021
where and m is the number of domains and slots, respectively. However, because each domain-slot pair is modeled independently, the relationship among the domain-slot pairs can not be learned. For example, if the user first asked for hotel in certain place and later asked for restaurant near that hotel, sharing the information between {hotel, orea) and (restaurant, area) would help the model recognize that the restaurant should be in the same area the hotel.
Recent approaches address these issues by modeling the dialogue state of every domain-slot pair in single run, given dialogue context (Chen al. 2020 Le al. 2019). This approach can be represented as follows:
(X,, ... Sm)
Because the encoder receives all of the domain-slot pairs, the model can factor in the relationship among the domain-slot pairs through the encoding process. For the encoder, these studies used models that are trained from scratch, without pre-training. However, since DST involves natural language text for the dialogue context, using pre-trained language model can help improve the encoding process. Several studies used BERT (Devlin et al.] 2019), pre-trained bidirectional lan- guage model, for encoding the dialogue context (Zhang et al. 2019 Lee et al.. 2019) Chao & Lane 2019 Gao et al. 2019). but did not model the dependencies among different domain-slot pairs. Our approach fills the gap between these previous studies. In this work, we propose a model for multi-domain dialogue state tracking that effectively models the relationship among domain-slot pairs using a pre-trained language encoder. We modify the input structure of BERT, specifically the special token part of it, to adjust for multi-domain DST.
The [C'LS] token of BERT (Devlin et al. 2019) is expected to encode the aggregate sequence rep- resentation as it runs through BERT, which is used for various downstream tasks such as sentence classification question answering. This [CLSS] token can also be used as an aggregate represen- tation for a given dialogue context. However, to store information for different domain-slot pairs at the same time. In this respect, we propose to use multiple special tokens, one for each domain-slot pair. Using a separate special token for each domain-slot pair is more effective in storing information for different domains and slots since each token can concentrate on its corresponding domain and slot. We consider two different ways represent such tokens: DS-merge and DS-split. DS-merge employs single token to represent single domain-slon pair. For example, to represent domain-slot pair of (restaurant, area). we use a special token DS (resttauuant
multi-domain dialogue, single [CLS] token has
DS-split, on the other hand, employs tokens separately for the do- main and slot and then merges them into one represent domain-sle slot pair. For {restaurant, area), the domain token Drestaurant and the slot token Sarea- is computed separately and then merged. We use { (DS) merge and (DS) split represent the special tokens for DSS-merge spectively. Unless
split absolutely necessary to specify whether the tokens are from DS-merge or DS-split, we'll refer the DS-produced tokens as (DS) tokens, without special distinction, in our descriptions forward. The {DS} tokens, after being encoded by the pre-trained language encoder along with the dialogue context, used predict its corresponding domain-slo slot value for given dialogue context.
DS-split,
Turs
L'tterances
Dialogue State
Tum
System:
Uset: am looking for place to that has cheap price range and should hotel
(botll, cheap). Chotel, type. hotel)
System:
Okay, you have specific you stay In? No, just nead to make sure cheip. and parking
(hotsl, mage, choap), (hotel type. hotel). [hotr), porking, yrs).
picc razge, cheap). Chotel type, botel), (botel, parking, yest; (hotel, book day, Toesday), (hotel, book people. 6). (hotel, book stog. 3)
Tum
found cheap howl you includes parking. Do you like me book Users Yes placee. 6 prople 3 rights starting on Tuesday
System:
Figure 1: An example dialogue and its dialogue state.
2

---

Under review as a conference paper at ICLR 2021
2 RELATED WORKS
Recent work on dialogue state tracking can be largely divided into two groups according to how the slot-values are predicted: fixed-vocabulary and open-vocabulary. The fixed-vocabulary approach, also known as the picklisted-based approach, uses a classification module to predict the dialogue state for each slot from pre-defined set of candidate values (Zhong al. 2018 Nouri & Hosseini- Asl 2018 Ramadan et al. 2018 Eric et al. 2019 Lee et al.) 2019 Chen et al., 2020). The open- vocabulary approach generates the dialogue state for each domain- slot pair either by using gener- ative decoder to generate text (Wu et al. 2019 Hossemm--As! ct al. 2020) or by extracting text spans from the dialogue history (Gao et al. 2019 Goel et al. 2019] Heck et al. 2020). There is also an approach to use both picklist-based and span-based methods according to the slot type (Zhang et al.) 2019).
For models that deal with multi-domain dialogue, how they deal with different domain-slot pairs another way to divide them. The first approach encodes the dialogue context independent of the domain-slot pairs and uses separate modules for each domain-sles pair (Eric et al., 2019 Gao al., 2019 Goel et al.; 2019 Heck et al. 2020). The second approach encodes the dialogue context the prefix and run the encoder multiple times (Nouri & Hosseini-AsI] 2018: Wu et al. 2019). Other approaches encode the dialogue context independently but merges it with domain-slot pair information later with separate fusion module (Zhong al., 2018 Ramadan et al. 2018: Lee et al. 2019). However, none of these models are able to model the relationship among different domain-slot pairs because there is module that enables the interac- tion between them.
using the domain-slot pair information
2019) and (Chen et al. 2020) directly models the relationship among different domain- slot pairs. (Le et al. 2019) uses Fertility decoder to learn potential dependencies across domain- slot pairs, but without using pre-trained language model. Also, their model requires additional data such as system action and delexicalized system responses for its performance. (Chen et al., 2020) also explicitly models the relationship among different domain-slot pairs by using a Graph Attention Network (GAT) (Veličković et al., 2018). Schema graphs, which is the relation graph between domains and slots, are utilized for connecting edges in the GAT. Our work different from these works in that we leverage the power of pre-trained language encoder for directly modeling the dependencies among different domain- pairs.
[Le et al.,
Hosseini-As) al. 2020 takes different approach from the others by using multi-task learning that encompasses DST as well as action and response generation with a generative language model GPT-2 (Radford al. 2019). However, since our work is focused on DST, we consider the model that is trained on DST only. In the decoding process, dialogue states for different domain-slot pairs are sequentially generated.
PROPOSED METHOD
(DSC) encoder, which Our model is composed of three parts. The first is the representing domain-slot pairs. Next is encodes the dialogue context along with the special tokens slot-gate classifier, which is preliminary classifier that predicts whether each domain-slot pair is relevant to the dialogue context. The adopted the concept of the slot-gate classifier from (Wu et al.) 2019) and made adjustments to apply to our model. The last is the slot value classifier for predicting the value for each domain-slot pair among the candidate values.
In the following descriptions, we assume dialogue context with total of T' turns. The task is to predict the dialogue state, which are [domain, slot, value} triplets for all domain-slot pairs, for every turn 1...... T, using the dialogue context until each turn. Section show the overview of our proposed model.
3.1 DOMAIN-SLOT-CONTTNE ENCODER
The main structure our model is the DSC encoder, which uses a pre- pre-trained language to encode the dialogue context along with (DS) tokens. For the pre- pre-trained language encoder, used ALBERT (Lan et al.) 2019) due to its strong performance on numerous natural language understanding tasks while having fewer parameters compared to other BERT-style encoders. (DS) tokens work like
3

---

Under review as a conference paper at ICLR 2021
Stot-gate classäeer
Slat-gate clnsslier
VDS, (65)
VDS, e L0S
Rip
Gatdoul
2
Sip
Siot-vaine classifier
production
Neo
Vi,,s, C. (00)
ALBERT
jendeziio Soo
ALBERT
CAS
CLS
Slot-valne clacsifler
CLS
Dyy
vng, (0rl
D5,0
Vularpyy
volve
Yuluead!
N
CLS
(a) DS-merge
(b) DS-split
Figure 2: Model overview.
the [CT.S] token for BERT, encoding information corresponding its domain-slot pair (DSS-mrrge) or domain and slot (DS-split). The set of special tokens for each layout are shown in Eq. (3) and Eq. 4 respectively. In DS-merge, we used special tokens for each individual domain-slot pair. If there are many domain-slot pairs, using this layout can increase the number of special tokens as each domain-slot pair requires a separate special token. In DS-split, we used separate tokens for the domain and slot. To represent domain-slot pair, we merged the corresponding tokens from each domain and slot by concatenating them. This promotes modeling compositionality, since same slot token can be used for different domains. These {DS} tokens and the dialogue context are processed through the DSC encoder, which results in each token {D.}} being encoded with contextualized representations according to its domain and slot.
(DS} merge
{DS (y ,DS (dsmamminn,))
(3)
(DS) split
(D domaingy: Ddomainga) Sslotay:" Satot())
(4)
Fig. 3] shows the input representation of the DSC encoder. The sequence begins with {DS} tokens. The special token [CLS] follows, which encodes the overall information of the dialogue context. For the dialogue context, we added a special token [SEPa] to separate each user system utterance, which is added at the end of each utterance from the user or system. The input ends with a special token []EP] the end-of-sequence token.
types of embeddings are summed up to represent each token embedding. We used the pre-trained word embedding of ALBERT, except for the {DS} tokens, which are randomly initialized. We introduced the token type embedding to differentiate the {DS} tokens, user utterances tokens, and system utterances tokens. For DS-merge, domain-slot pair, whereas for DS-splir, we used two token type embeddings, one for the domain and the other for the slot. We did not apply this embedding for the [CLS] token. Position embeddings are also employed from ALBERT, but the index of the positional embedding starts from the [CLS] token. We did tokens is meaningless. Lastly, the segment embedding from ALBERT was used to represent the whole sequence as a single segment, which is the default segment embedding of ALBERT.
used single token type embedding represent
the positional embedding for the (DS) tokens the order within those
DSC encoder encodes contextualized embeddings for every input token. However, for the slot- gate classifier and slot-value classifier, we only use the special token outputs of the DSC encoder ([CLS] token and (DS) tokens). This is formally defined as follows for DS-merge and DS-split, respectively, for turn
DS DS (m)m CLS DDCeencoder merge CISSX',,SEP D,. Sm, CLS = DSC (( spli, CLS, (SSS)),
(5)
(6)
where X represents the dialogue context of (S', SSEP,U' .SP.,.... "St SS,SEP, U², U",S)P,). and Sth represents the utterance for the th turn for the and system respectively. The (DS)
4

---

Under review as a conference paper at ICLR 2021
domsin slot takens
user utterance
system unterance
Input
Dr
CLS
would Like a tati from New York Boston
SBR
What time do want to lerve?
SER.
7:00
SEP
Word embodking
Foss
Fossuu
Feus
Buzt
Espp,
Eucre
Eprzu
Facty
Eyer
+
Tokan type
emboodding
Fas
Eyer
Eggy
Ruur
+
embedding
Segment
embodking
Exzpi
Figure 3: Input representation for the DSC encoder. The example here shows a dialogue context for 2 turns of (8), U S U²). S is omitted because the sequence starts with the user utterance and S' is just a placeholder: blank sentence. In this figure, the special token layout for {DS) tokens represented in DS-merge. Eutt, represents the sequence of word embeddings for each utterance after tokenization.
tokens and [CT.S] token with the hat notation represents the encoded output of the DSC encoder for those special tokens. They are vectors of Rd, where d is the hidden dimension of ALBERT.
3.2 SLOT-GATE CLASSIFIER
For the slot-gate classifier, use the DSC encoder output of the {DS) tokens for each domain- slot pair to predict whether it is relevant to the dialogue or not. In previous methods, gating used categories of (prediction, dontcore, none), where prediction means slot value is not dontcore or none and dontcare means that the predicted slot value is dontcare and none means that the domain-slot is non-relevant. The label for slot-gates are made from the slot-values. However, the performance for the dontcare category was far inferior to the other two categories, so we dismissed the dontcare category and only used (prediction, none]. In our preliminary models with ALBERT large-12, the prediction and recall for dontcare was 48.87% and 17.21 %,, respectively. The preci- sion and recall for none showed 98.91% 1,, 99.45% and prediction 96.16%, 94.93%, respectively. In this setting, the donlcore category is included in prediction. For DS-merge, the slot-gate classifier predicts the value using the domain-slot pair special token. For the domain-slot pair domain and slot j. the slot-gate classifier output for DS-merge is
Gatep.s, sigmoid ( WGns DS
where Wan,s E Elid, For DS-split, the slot-gate classifier uses the concatenated output of the cor- responding domain and slot token. Similarly, for the same domain-slot pair, the slot-gate classifier output for DS-split is
Gatep,s, = sigmoid(Werr (D..s
D. )
(8)
where represents concatenation of vectors and W WGPDes)) classification is as follows.set
RR22d The loss objective for the gate
Cyate Σ
BinaryCros:Entrpp( Gatep.s;).
(9)
(L.))
where DS refers to the set of all domain-slot pairs and y is the binary slot-gate label for domain and slot j. If the domain-slot is predicted to none, the corresponding output of the slot-value classifier is changed into none regardless of the prediction of the slot-value classifier.
5

---

Under review as a conference paper at ICLR 2021
3.3 SLOT-VALUE CLASSIFIER
We employ the excd-vocabulary based classification method for predicting slot values. As in Zhang al. 2019), the candidate-value list for each domain-slot pair was constructed by using the values from the training dataset, rather than using the incomplete ontology from the dataset. The [CLS] token concatenated with each token from {DS), and used as the imput the slot-value classifier for each domain-slot pair. The slot-value classifier output of domain i and slot j for DS-merge is as
follows:
V,l = softmar (D. CLS)),
(10)
where WVnsa. E R x 2d and RD,s, the number of candidate values for domain i and slot Similarly, for DS-split, the slot-value classifier output is
Valuep.s, = softmax (Wv.o, (LS,)
(11)
where s,,,,,,) E R x 3d. The loss objective for the slot-value classification is as follows:
Laluue
Σ
CrossEnlropulub..s Valuep.s;),
(12)
(SJ)EDS
where VD.S; is the label for domain and slot j.
3.4 TOTAL OBJECTIVE FUNCTION
The DSC encoder, slot-gate classifier and slot-value classifier is jointly trained under total ob- jective function below.
Crotal Logate Loaauue
(13)
EXPERIMENT SETUP AND RESULTS
We evaluate our model using the joint goal accuracy, which considers a model prediction to be correct when the prediction jointly matches the ground truth values for all domain-slot pairs, given a dialogue context.
4.1 DATASET
We the MultiWO--2.1 (Eric et al. 2019) and MultiWO--2.2 dataset (Zang al. 2020), both of which fixed noisy annotations and dialogue utterances of the MultiWOZ 2.0 dataset Budzianowski al. 2018). The dataset contains domains and over 10,000 dialogues. We follow the previous studies and use 5 domains (train, restaurant, hotel, taxi, attraction) with 30 domain-slot pairs. The other two domains (police, hospital) have little data and do not appear in the test dataset. For MultiWOZ-2.1, follow the pre-processing explained in (Wu al. 2019). For MultWWOZ-2.2, we use the raw data as given without any pre-processing.
4.2 SETUP
For the pre-trained language encoder, we used ALBERT (Lan et al. 2019) from HuggingFace (Wolf et al. 2019) in Pytorch (Paszke et al., 2019). We used the xxlarge-v2 version of ALBERT for the main experiment and compare other versions (base-v2, large-v2) compared ERBa (Liu et al.) 2019) to generalizability of our model. The optimizer was AdamW [Loshchloov & Hutter 2018) with learning rate 1 for ALBERT-xlarge-v2, ALBERT-xxlarge- v2 and RoBERTa-lary and 5e for ALBERT-base-v2 ALBERT-large-i and RoBERTa-baue We applicd linear warm-ur up followed by linear decay for the learning rate. We trained all models with the effective batch size of 32, using gradient accumulation for bigger ALBERT models. Models were
the analysis section. We also

---

Under review as a conference paper at ICLR 2021
Table 1: Results for the test dataset of the Multi-WOZ 2.1 and MultiWOZ. 2.2 dataset. *; extra
supervision information is from (Hosseini-As) et al. 2020).
Model
Extra Supervision
MultiWOZ-211
MultiWOZ2.22
SGD-baseline (Rastogi et al. 2020
43.4
42,0
TRADE Wu el al. 2019
46,0
45.4
NADST
Le al 2019
sys. action, delex. sys, response
49.04
DSTQA Zhou Small 2019
knowledge graph*
51.17
DS-DST [Zhang el al.
51.20
$1.7
DS-Picklist (Zhang
2019
53.30
SST (Chen et al.
2120
55.23
TripPy (Heck el al. 2020
action d
55.3
SimpleTOD (oopll (Hoscini-AS Asl ct al.
2020
55.76
CHAN (Shari et al. 20220
58.55
ConVBEKI-DG+ Muiti (Mehri et 2020
DialoGLUE Mchri et al. 2020 data
58.7
work (DS-merge, ALBEKI-xxtorg--z)
69.23
77.28
Our work (DS-split, ALBERT-cslrgg-22)
67.31
82.23
selected based on their joint goal accuracy on the validation data split. Only the training data was used build the labels for each domain-slot pair. We used two NVIDIA V100 for our training. The original ALBERT was pre-trained with sequence length of up to 512 tokens. However, dialogues that are longer than 512 tokens exists in the data. Usually, the standard procedure for this situation to truncate the sequence up 512 tokens and discard the remaining tokens. However, to cover dialogues longer than 512 tokens that are in the dataset, we resized the positional embedding to cover maximum length of the dialogue. We preserved the original pre-trained position embedding for positions indices up to 512 and randomly initialized the remaining position indices. This method showed better results than limiting the maximum sequence length to 512. We plan to release code on Github.
4.3 RESULTS
Table shows the joint goal accuracy of model compared to previous methods. Both of our models show better performance among models without any additional supervision other than the dialogue context and domain-slot pair labels. Especially, the DS-split, ALBERT-xxlarge-V version of our proposed model achieves state-of-the-art result on the MultiWOO-2211 and MultiWOO-2.22 dataset, without any form of extra supervision. However, in smaller models, The model with DS- split shows better results than the model with DS-merge. This shows that in models with enough capacity, the slot-sharing of DS-split was more effective. However, this was not the case for smaller ALBERT models, which is explained in Section 4.4.2 This is important in that scalability is much better for DS-xplit than DS-merge, as many slots can be shared across different domains, reducing the number of special tokens to be used. We show the individual domain-slor accuracy in Appendix A.2 Table 4
4.4 ANALYSIS
In this section, we show that relationship modeling among different domain-slot pairs is indeed the key factor of our proposed model by running ablation studies. Also, we compare the effect of the size and type of the pre-trained language encoder in terms of performance.
4.4.1 RELATIONSHIP MODELING AMONG DIFFERENT DOMAIN- SLOT PAIRS
First, we did not use any { (DS} tokens and only used the CLS token. Because there are no dedicated special tokens for each domain-slot pair, the performance is very poor shown in 'None' row in Table 2 This shows that our approach to introduce (DS} is effective.
Next, evaluate the effect of relationship modeling among different domain-slot pairs, we blocked the attention among different {DS} tokens during the encoding process, which restricts direct in- teraction among {DS} tokens. Table shows that without the relationship modeling, model performance deteriorates by a substantial amount. This validates our idea that relationship modeling the crucial factor for approach.

---

Under review as a conference paper at ICLR 2021
Table 2: Results for ablation of domain-slot relationship modeling the test dataset of MultiWOZ.
2. 2.
Pretrained Language Encoder
{DS} token layout
Joint Goal Accuracy
None
45.49
DS- DS-merge
55.48
ALBERT-large-v2
w/o relationship modeling
53.38
DS-split w/o relationship modeling
55.06
52.84
None
50.96
DS-merge w/o relationship modeling
67.31
ALBERT--xlarge--2
63.46
DS-split
69.23
w/o relationship modeling
61.54
Table 3: Results for different ALBERT configurations on the evaluation test dataset of MultiWOZ 2.1.
(DS) token layout
Pretrained Language Encoder
Joint Goal Accuracy
ALBERT-base-v2
55.01
ALBERT-large-v2
55,48
ALBERT-xlarge-V2
57.02
DS-merge
ALBERT-cularge-12
67.31
RoBERTa-buse
53.85
RoBERTa-large
55.77
ALBERT-base-v2
53.54
ALBERTTlarge-v2
55.06
ALBERRY-xlrge--2
55.51
DS-split
ALBERT-uxlarge-22
69.23
RoBERTa-base
57.69
RoBERTa-large
58.65
In the Appendix A. we show some examples of wrong predictions that models without direct relationship modeling has made.
4.4.2 SIZE AND TYPE OF THE PRE-TRAINED LANGUAGE ENCODER
We compared ALBERT and RoBERTa Liu et al. 2019) and various model sizes within those pre- trained language encoders. Table shows the result for different versions of the pre-trained language encoders. For ALBERT, bigger language model shows better results is shown in various down- stream tasks that ALBERT was evaluated (Lan al., 2019). Except for ALBERT-xX-large all other configurations show that DS-merge shows better performance than DS-split. Based on the dras- tic increase in performance with xx-large, we presume that the high model complexity of ALBERT: xx-large enabled (DS) sptit tokens to effectively encode information and make slot-sharing to work. In smaller models, this slot-sharing might not have been as effective due to their smaller encoding capacity. Also, concatenation, which was used for merging domain and slot embeddings in DS DS-split, might not have been enough for fully representing the information for the domain-slo pair in smaller models. RoBERTa also shows similar results with bigger models showing stronger performance.
4.4.3 LEARNING CURVE
Fig. [ shows the learning curve of the ALBERT-xxlarge-22 on the MultiWOZ-2.2 dataset. The joint goal accuracy steadily increases after the slot-value loss plateaus.
8

---

Under review as a conference paper at ICLR 2021
100.0
DS-split
10.0
DS-merge
80.0
D.--DS-gge DS-split
8.0
DD--merye
Jeizz Goal God Acauncy
7.0
60.0
6.0
5.0
Slot
40.0
3,0
20.0
1.0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
Training Step
Figure 4: Joint goal accuracy and slot-value loss versus training steps on the evaluation dataset of MultiWOZ-2.2. ALBEKT-xxlarge-P2 was used for the DSC-encoder. Straight lines indicate the joint goal accuracy and dashed lines indicate the slot-value loss. The x-axis indicates the training steps.
5 CONCLUSION
In this paper, propose model for multi-domain dialogue state tracking that effectively models the relationship among domain-slot pairs using a pre-trained language encoder. We introduced two methods represent special tokens for each domain-slot pair: DS-merge and DS-split. These tokens work like the [CLS] token for BERT, encoding information corresponding to its domain-slot pair (DS-merge) or domain and slot (DS-split). These special tokens are run together with the dialogue context through the pre-trained language encoder, which enables modeling the relationship among different domain-slot pairs. Experimental results show that our model achieves of-e-off-hhe-n per- formance on the MultiWOZ-2.1 and MultWWO--2.2 dataset. The ablation experiments show that relationship modeling among different domain-slot pairs is the key element of our model. Also, we showed that larger pre-trained language encoders improves performance. We hope to advance our research by finding ways to effectively apply our model towards the open-vocabulary approach, which will enable better generalization for candidate values that outside of the training data.
ACKNOWLEDGMENTS
This research was supported and funded by the Korean National Police Agency, [Pol-Bot Develop- ment for Conversational Police Knowledge Services PR09-01-000-20)
REFERENCES
Pawel Budzianowski. Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016-5026. 2018
Guan-Lin Chao and Ian Lane. Bert-dst: Scalable end-to-end dialogue state tracking with bidirec- tional encoder representations from transformer. arXiv preprint arXiv:1907.0300 2019.
Lu Chen, Boer Lv, Chi Wang, Su Zhu, Bowen Tan, and Kai Yu. Schema-guided multi-domain dialogue state tracking with graph attention neural networks. In AAAI, pp. 7521-7528, 2020.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019.

---

Under review as a conference paper at ICLR 2021
Mihail Eric. Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani- Tur. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint 1907.1:9900000000 2019.
Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Hakkani-Tur. Dialog state tracking: A neural reading comprehension approach. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pp. 264-273. 2019.
Rahul Goel, Shachi Paul, and Dilek Makkani-Tünh Hyst: A hybrid approach for flexible and accurate dialogue state tracking. Proc. Interspeech 2019, pp. 1458 1462, 2019.
Michael Heck, Carel van Nickerk, Nurul Lubis, Christian Geishauser, Aisien-Chin Lin, Marco Moresi, and Milica Gašié. Trippy: A triple copy strategy for value independent neural dialog state tracking. arXiv preprint arXiv:2005.0255 2020.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for task-oriented dialogue. arXiv preprint arXiv:2005.0079, 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2019.
Hung Le, Richard Socher, and Steven CH Hoi. Non-autoregressive dialog state tracking. In Inter- national Conference on Learning Representations, 2019.
Hwaran Lee, Jinsik Lee, and Tae- Yoon Kim. Sumbt: Slot-utterance matching for universal and scalable belief tracking. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics. pp. 5478-5483, 2019.
Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoeer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining 1692, 2019.
approach. arXiv preprint
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018
Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. Dialoglue: A natural language understanding benchmark for task-oriented dialogue. arXiv preprint arXiv:2009.1335 2020.
Elnaz Nouri and Ehsan Hosseini-Asl. Toward scalable neural dialogue state tracking model. arXiv preprint 10/2:v8//22000000 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in neural information processing systems, pp- 8026-8037, 2019.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Osman Ramadan, Pawet Budzianowski, and Milica Gašić. Large-scale multi-domain belief tracking with knowledge sharing. arXiv preprins arXiv: 1807.86077577 2018.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8689-8696, 2020.
Yong Shan, Zekang Li, Jinchao Zhang, Fandong Meng, Yang Feng, Cheng Niu, and Jie Zhou. A contextual hierarchical attention network with adaptive objective dialogue state tracking. arXiv preprint arXivv2006.075: 2020.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
10

---

Under review as a conference paper at ICLR 2021
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State- of-the-art natural language processing. ArXiv, 1b0//191..3 2019.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pas- cale Fung. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 808-819, 2019.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. Multiwoz 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. ACL 2020, pp. 109, 2020.
Jian-Guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wan, Philip S Yu, Richard Socher, and Caiming Xiong. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. arXiv preprint arXiv:1910.03544, 2019.
Victor Zhong, Caiming Xiong, and Richard Socher. Global-locally self-attentive encoder for dia- logue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 1458-1467, 2018.
Li Zhou and Kevin Small. Multi-domain dialogue state tracking as dynamic knowledge graph en- hanced question answering. arXiv preprint arXiv:1911.06192 2019.
A
APPENDIX
A.1 RELATIONSHIP MODELING EXAMPLES
A.1.1 EXAMPLE
Fig. shows an example of wrong prediction that the model without domain-slot relationship modeling makes. The value for {taxi, departure) is not explicitly mentioned in the dialogue context. However, Our full model correctly predicts the value for [laxi, departure), which can be inferred from the dialogue context and (hotel, name). However, the model without relationship modeling fails to predict the correct value for [taxi, departure).
User: am staying in cambridge soon and would like to stay a and b guest house.
System: sure, how many days and how many people?
User: we are staying people for nights starting from tuesday. need the reference number
System: your booking is successful! your reference number is iigraOmi. do you need anything else?
User: yeas, what recommend if want see good architecture in the west part of town?
System: unfortunately there is no good architecture on the west end but can look in other parts of town if you want
User: what about museum?
System: what part of town there are none in the west.
User: there are no museums in the west at all?
System: sorry about that, there actually 7 in that area.
User: great, can get the postcode, entrance fee and address of 1 of them?
System: cafe jello gallery has free entrance fee. the address is cafe jello gallery, 13 magdalene street and the post code cb30af. can help you with anything else?
User: yes please. need taxi commute.

---

Under review as a conference paper at ICLR 2021
System: when would you like leave and arrive?
User: would like to get to the gallery by 13:45, please.
System: sure, lookout for blue volvo the contact number is 07941424004 can help with anything else?
User: that is all for now. thank you so much
Domain-slot
Proposed method
w/o relationship modeling
hotel-name
and b guest house
a and guest house
hotel-book day
tuesday
tuesday
hotel-book day
4
4
attraction-area
Tuesday
Tuesday
attraction-type
west
west
taxi-destination
museum
museum
taxi-departure
cafe jello gallery
none (WRONG)
taxi-arriveby
13:45
13:45
Figure An example of dialogue and its dialogue state.
A.1.2 EXAMPLE 2
Fllso shows an example of a wrong prediction that the model without domain-slot relationship modeling makes. The value for {train, day} is not explicitly mentioned in the dialogue context. In similar manner from the example above, can be referred from the {restaurant, book day).
User: would like to find a particular restaurant in cambridge. the name of the restaurant is restau- rant 2 two. could you give me the location?
System: restaurant 2 two is nice french restaurant located at 22 chesterton road chesterton. would like me to book you table?
User: that would be great. i need it for 8 on friday.
System: do you have time preference?
User: yes at 1 11:15 if that is not available can do 10:15
System: the booking for 10:15 was successful they will reserve the table for 15 minutes. the refer- ence number is 6b527vj5.
User: thanks. you help find train, too? want leave cambridge time after 12:15.
12

---

Under review as a conference paper at ICLR 2021
Domain-slot
Proposed method
w/o relationship modeling
train-day
friday
none (WRONG)
train-departure
cambridge
Cambridge
train-leaveat
12:15
12:15
restaurant-name
restaurant 2 two
restaurant 2 two
restaurant-book
10:15
10:15
time
restaurant-book day
friday
friday
restaurant-book
8
people
Figure An example of dialogue and its dialogue state.
A.2 INDIVIDUAL SLOT ACCURACY
Table shows the individual domain-slor accuracy for the ALBERT-xxlarge-12 model on the MultWWOZ-2.2 dataset.
13

---

Under review as a conference paper at ICLR 2021
Table 4: Individual domain-slot accuracy for the ALBERT-xxlarge--2 model the MultiWO7-2.2
dataset.
Domain-slot
DS-merge
DS-split
attraction-area
94.56
96.81
attraction-Nn name name
94.43
96.62
attraction-tyre -yype
94.4
96.92
hotel-area
95.41
97.75
hotel-book day
96.05
98.5
hotel-book people
96.32
98.52
hotel-book stay
95.97
98.5
hotel-internet
95.6
97.76
hotel-name
94.84
97.21
hotel-parking
95.7
98.1
hotel-price range
95.36
97.72
hotel-stars
94.96
97.34
hotel-type
95.47
97.35
restaurant-area
94.81
97.21
restaurant-book day
97.72
100
restaurant-book people
97.61
100
restaurant-book time
97.67
100
restaurant-food
95.1
97.21
restaurant-name
94.12
96.34
restaurant-price range
94.85
97.24
taxi-arrive by
97.19
99.42
taxi-departure
96.29
98.57
taxi-destination
96.63
98.73
taxi-leave at
97.05
99.28
train-arrive by
97.54
100
train-book people
95.49
97.78
train-day
93.03
95.14
train-departur
92.6
95.01
train-destination
92.73
94.84
train-leave at
94.87
97.08
14


Reviews:
[{"review_id": "n0_EQ7h4Kkh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper showcases how pre-training can help with Dialogue State Tracking. The authors explicitly model\nthe relationship between domain-slot pairs. With their encoding and using strong pre-trained initializations\nthey are able to improve the joint goal accuracy by almost 1.5 points which is impressive.\n\nReasons for score: \nThis is a very well written paper and will be a good resource for people working on the task of Dialogue State Tracking.\nThe authors show how they can model relationships between domain-slot pairs and how they can encode them effectively\nusing pre-trained representations.\nI am hoping that the authors can address some of the cons during the rebuttal period.\n\nPros:\n1. Good dialogue representation which helps with the task of state tracking\n2. Simple model consisting of encoders and 2 classifiers which are well explained.\n3. Clear ablation study showing the value of 1) pre-training and 2) modeling relationship between domain-slot values\n\n\nCons:\n1. This approach, like other popular approaches, suffers from the problem of having a fixed output vocabulary for slot values - hence limiting its scalability. While this cannot be addressed in this work, this is a drawback of this approach.\n2. Some of the design decisions are stated but not well explained\n- Only one pre-training method compared\n- Authors mention they drop \"dontcare\" from slot gating but don't show the affect with or without it.\n- Not much details on the setup and how it was trained.\n3. Not much qualitative analysis.\n\nPlease address and clarify the cons above \n\nTypos/Areas for improvement:\n1. Section 3.2 and 3.3 can be shortened a lot. I would suggest showing more analysis.\n- More examples of type of  mistakes fixed.\n- Which turn in the dialogue does the error decrease the most.\n- How much is the training time/ accuracy tradeoff\n2. Adding another layer to make DS-split work should be trivial, there is no reason to leave that to future work.\nCould you show how the results look with that?\n\n------\n\nUpdating score based on authors' response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "l2wbpyOZt7H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposed a new approach for modeling multi-domain dialogue state tracking by incorporating domain-slot relationship using a pre-trained language encoder. The proposed approach are based on using special tokens to mode l such relationship. Two kinds of special tokens are proposed to represent domain-slot pair, DS_merge token for each specific pair, and tokens for every domain and slots separately \n\n\n\nComments:\n\n1- what is the role of segment embedding (Fig3) for DST tank? Are two different segment used in the pretraining of model?\n\n2- Table 1:\n                  2-1 what type of label cleaning is used to compute joint goal accuracy? From SimpleTOD paper, each baseline has used different label cleaning\n                  2-2: DS-split is bolded, while it is lower than SimpleTOD\n\n3- Table 2: the results indicate that DST performance drops in total by blocking attention across different DS tokens. However, it is not clear how much of the performance drop belongs to turns with cross-domain related slots. The figure 4 only present one example of this case, which might not be correct for all wrong predictions. Also, it is helpful to report DST for single domain and to evaluate the importance of proposed approach. \n\n4- Since the proposed approach can be used on any pretrained encoder, the evaluation on BERT and/or Roberta is helpful to understand the robustness of approach to the choice of pretrained encoder.\n\n\n\n--------------------------------------------------------------\nPost rebuttal: \n\nTable 3:\nthe results indicates that only Albert-xxlarge achieve very high performance (222M). however, comparable models to other approaches, such as Roberta-base or albert-xlarge achieved around ~57% performance which is within margin of previous arts. for example, SimpleTOD with gpt2-base (124M) achieved 55.7% and ConvBert achieved 58%.\nTherefor, it is unclear why Albert-xxlarge get so much higher performance compared to other encoders, since same tokenization and domain-slot relation is used. \nBased on results in Table 3, there is inconsistency in which domain-slot relation does not always results in better performance,  and it depends on the choice of encoder too. \n\nOverall, the proposed architecture is very similar to TRADE model, in terms of using an encoder for dialogue history, slot-gate and slot-value classifier. The only difference is in using a much powerful pretrained encoder. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "JGicoZ1Rsmx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nIn this paper, the authors proposed a multidomain state-tracking model that leverages the relationship among different domain-slot pairs. This is done by leveraging the full-attention step over the [CLS] special token and by providing all the domain-slot pairs as a special token to a pre-trained language model (Figure 2 is very clear). To predict the value of the slot $D_{i,j}$, the author concatenates the representation of the [CLS] token, share among all the domain-slots, and the $D_{i,j}$, provided as input, and use a gating mechanism, by only using $D_{i,j}$ representation, to decide whether require as value (i.e., prediction) or not (e.g. None). \\\n\nThe authors experimented using ALBERT (Lan et al., 2019) as a pre-trained language model, on the well-known benchmark MultiWoZ 2.0  (Budzianowski et al., 2018)  2.1 (Eric et al., 2019). The authors studied different format to represent $D_{i,j}$ DS-merge (i.e., one token per domain-slot) and DS-split (i.e., one token per slot and one per domain, thus more scalable). The reported performance is state-of-the-art at the time of submission. \n\n[Pros]\n- The paper reads well and it is easy to follow for people working on Task-Oriented dialogue.\n- The proposed method is simple and effective, and it would be easy to reproduce. \n\n[Cons]\n- The idea of using domain-pairs as input to a large pre-trained model is not novel (Wu et al., 2019; Zhang et al., 2019; Lee et al., 2019), as also pointed out by the authors, but the authors do not explicitly clarify this in the methodology section, leading the reader to believe that the domain-pairs is their own contribution. Same for the slot-gate (Wu et al., 2019)\n- The authors claim to learn relations between slots, but the analysis section is very thin and it just shows an ablation by masking the attention between the slot. Two points: why not just removing the [CLS] token instead of removing the attention, and why just using on ALBERTA large. For instance, the authors said \"For this experiment, we used the ALBERT configuration of large-v2,\nfor faster experimentation\" which is contradictory since large-v2 is the slowest to run I guess. Can the authors show this ablation for all the model size? \n- Although, MWoZ is the current benchmark for DST in ToDs, there are also other datasets for this task that can be considered (e.g., Schema Guided Dialogue (SGD) (Rastogi et.al. 2019))\n\n\n[Reason to Reject]\nThe main contribution of this paper is very thin, adding the [CLS] token as input, and the main technical contribution is not well explored (missing an in-depth ablation). \n\n[Reason to Accept]\nState-of-the-art performance at the submission time. To be noted, (Mehri et.al. 2020) reported better performance in MWoZ and other datasets, but this paper was released after the ICLR submission deadline.\n\n[Question]\n-  Can the authors show the ablation for all the model size? \n\n[Suggestion]\n- Figure 4 is very hard to read. I suggest to better format the dialogue. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "A49FqJHZEG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros\n•\tThis paper incorporates the multi-domain domain-slot pairs into the bert input so that the relations between sentences, domain-slot are modeled.\nCons\n•\tIt’s better to experiment on more dataset to prove the method’s effectiveness.\nComments\n•\tThis paper   https://arxiv.org/abs/2006.01554 seems get better performance on the same dataset.\n•\tIn the domain split/merge method, does the domain/slot used as vocabulary tag or word embedding?\n•\tThe slot-gate classifier’s output is fed into the slot-value classifier, how does this affect the performance?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors explore modeling the relationship between domain-slot pairs in multi-domain dialogue state tracking via use of special tokens in pre-trained contextualized word embeddings (i.e., one special token for each domain-slot pair or special tokens for the domain and the slot that are merged). Beyond this, the basic architecture is very similar to the TRADE architecture (and papers that build on this general slot-gate + slot-value classifier) for the fixed vocabulary setting. Experiments are conducted on the MultiWOZ 2.1/2.2 datasets, demonstrating impressive improvements over recent results.

== Pros ==
+ They demonstrate that domain-slot interdependencies can be modeled through special tokens for use with pre-trained embeddings.
+ The top-line empirical results are impressive.

== Cons ==
- Lack of a deep dive on the empirical analysis to show precisely why/where the proposed method is working better than existing work.
- The methodological advance is minimal beyond using better pre-trained embeddings.
- Only one dataset when others exist and this is largely an empirical paper.
- The writing is rushed and reads like a 'late-breaking' paper.

Evaluating along the specified dimensions:
* Quality: The quality of the work was the primary concern of the reviewers. Specifically, this reads like a 'late breaking' paper where the table of results is impressive, but there isn't significant examination of the empirical results showing why/when it works relative to competing methods. Focusing just on Tables 2 & 3, much of the improvement is ostensibly really due to the more powerful embeddings. Contextualizing this wrt {SimpleTOD, TRADE, DSTQA, Picklist}, this appears a minor methodological innovation centered around the input embeddings. The empirical results are impressive, but may very well be a result of the more powerful pre-trained embeddings -- additional empirical analysis and discussion might be able to convince the reader otherwise, but is lacking here.
* Clarity: This is a very simple idea, so it should be easily understood by most familiar with the research area. That being said, the paper seems very rushed in general.
* Originality: This applies ideas used in many NLP applications to the dialogue-state tracking problem. As previously stated, the architecture is similar to several existing DST formulations -- where the core idea is to model slot-value interdependencies through the contextualized embeddings using special token. While not a trivial idea, it also is something that many could/would have put together. Until it is abundantly clear that this isn't really a study of how to apply larger pre-trained embeddings to DST problems, it isn't clear that this is a significant dialogue systems advance beyond the strong performance.
* Significance: As stated, this isn't a significant methodological advance. However, the empirical results appear very impressive -- although the reviewers expressed some concerns regarding the evaluation. Since this is largely empirical, one of the reviewers pointed out that additional relevant datasets now exist, which would significantly strengthen the case.

In summary, the empirical results appear impressive, ostensibly setting the SoTA. However, there were several concerns regarding the novelty of the approach, if it is actually working better due to the reasons stated, sufficient analysis of the empirical results, amongst other things. Thus, despite the impressive results, the consensus evaluation was that this work is not ready for publication in its current form (even if the top-line results should be disseminated).

Decision: Reject