Non-Autoregressive Dialog State Tracking
Authors: Hung Le
Richard Socher
Steven C.H. Hoi
Venue:  (2020)
Keywords: task-oriented
dialogues
dialogue state tracking
non-autoregressive

Abstract: Recent efforts in Dialogue State Tracking (DST) for task-oriented dialogues have progressed toward open-vocabulary or generation-based approaches where the models can generate slot value candidates from the dialogue history itself. These approaches have shown good performance gain, especially in complicated dialogue domains with dynamic slot values. However, they fall short in two aspects: (1) they do not allow models to explicitly learn signals across domains and slots to detect potential dependencies among \textit{(domain, slot)} pairs; and (2) existing models follow auto-regressive approaches which incur high time cost when the dialogue evolves over multiple domains and multiple turns. In this paper, we propose a novel framework of Non-Autoregressive Dialog State Tracking (NADST) which can factor in potential dependencies among domains and slots to optimize the models towards better prediction of dialogue states as a complete set rather than separate slots. In particular, the non-autoregressive nature of our method not only enables decoding in parallel to significantly reduce the latency of DST for real-time dialogue response generation, but also detect dependencies among slots at token level in addition to slot and domain level. Our empirical results show that our model achieves the state-of-the-art joint accuracy across all domains on the MultiWOZ 2.1 corpus, and the latency of our model is an order of magnitude lower than the previous state of the art as the dialogue history extends over time. 

Published as a conference paper at ICLR 2020
NON-AUTOREGRESSIV DIALOG STATE TRACKING
Hung Le:; Richard Socher Steven C.H. Hoit
i Salesforce Research
(rsocher, , shoi shoij@ssaesforrs wesalesforreee...ooooooo.oooesee com
I Singapore Management University
hungle. 2018@phdcs smu. edu. sg
ABSTRACT
Recent efforts in Dialogue State Tracking (DST) for task-oriented dialogues have progressed toward open-vocabulary or generation- approaches where the models can generate slot value candidates from the dialogue history itself. These approaches have shown good performance gain, especially in complicated dia- logue domains with dynamic slot values. However, they fall short in two aspects: (1) they do not allow models to explicitly learn signals across domains and slots to detect potential dependencies among (domain, slot) pairs; and (2) existing models follow auto-regressive approaches which incur high time cost when the dialogue evolves over multiple domains and multiple turns. In this paper, we propose a novel framework of Non-Autoorerressive Dialog State Tracking (NADST) which can factor in potential dependencies among domains and slots to optimize the models towards better prediction of dialogue states complete set rather than separate slots. In particular, the non-autoress oon-autorerressive nature our method not only enables decoding in parallel to significantly reduce the latency of DST for real- time dialogue response generation, but also detect dependencies among slots at token level in addition slot and domain level. Our empirical results show that our model achieves the ofte-of--the-at joint accuracy across all domains on the MultiWOZ 2.1 corpus, and the latency of our model is an order of magnitude lower than the previous state of the art as the dialogue history extends over time.
INTRODUCTION
In task-oriented dialogues, a dialogue agent required to assist humans for one or many tasks such as finding a restaurant and booking hotel. As sample dialogue shown in Table D each user utterance typically contains important information identified as slots related to a dialogue domain such attraction-area and Irain-day. A crucial part of task-oriented dialogue system is Dialogue State Tracking (DST), which aims to identify user goals expressed during a conversation the form of dialogue states. A dialogue state consists of set of (slot, value) pairs e.g. (atrracton-area, centre) and (train-day, tuesday). Existing DST models can be categorized into two types: fixed- and open-vocabulary. Fixed vocabulary models assume known slot ontology and generate a score for each candidate of (slot, value) (Ramadan et al. 2018 Lee et al. 2019). Recent approaches propose open-vocabulary models that can generate the candidates, especially for slots such entity names and time, from the dialogue history (Lei et al. 2018 Wu et al. 2019).
Most open-vocabulary DST models rely on autoregressive encoders and decoders, which encode dialogue history sequentially and generate token 4; of individual slot value one by one conditioned on all previously generated tokens f1] For downstream tasks of DST that emphasize on low latency (c.g. generating real-time dialogue responses). auto-regressive approaches incur expensive time cost as the ongoing dialogues become more complex. The time cost is caused by two major components: length of dialogue history i.e. number of turns, and length of slot values. For complex dialogues extended over many turns and multiple domains, the time cost will increase significantly in both encoding and decoding phases.
Similar problems can be seen in the field Neural Machine Translation (NMT) research where a long piece of text is translated from one language to another. Recent work has tried to improve the
All work was done while the first author was research intern Salesforce Research Asia.

---

Published as a conference paper at ICLR 2020
Human:
visit theater the center of
Dialog State;
(attruction areu, centre), (altraction type. theatre)
System:
there motches do not have any info the lees. do you have any other preferences"
Human:
no other preferences.. . just want to sure to get the phone number of whichever theatre we pick.
Dialog State:
(artmction-aree centre), (attraction-yype theatre)
System:
recommend cambridge com exchange there phone number is 01223157851 is there anything else can help you with
Human:
am looking for miesday train.
Dialog State:
(attruction area, centre), (altraction name, the cambridge corn exchange). (attraction--yp) theatre), (train (traindday, tuesday)
System:
where will be departing from and whalls a your destination ?
Human:
from cambridge london liverpool street.
Cattraction-area centre), (altraction the cambridge exchange). (attraction-sype, (heatre). (train-day, tuesday).
Dialog State:
(traindopartuuc, cambridge), (train-destimation. london liverpool stroct)
Table 1: A sample task lusk-oriented dialogue with annotated dialogue states after each user turn. The dialogue states in red and blue denote slots from the attraction domain and train domain respectively. Slot values are expressed in user and system utterances (highlighted by underlined text).
latency in NMT by using neural network architectures such as convolution (Krizhevsky et al. 2012) and attention (Luong et al. 2015). Several non- and sett-autoee approaches aim to generate tokens of the target language independently (Gu et al. 2018 [ee et al. 2018 Kaiser et al. 2018).
Motivated by this line of research, we thus propose non-autoregress oonreusoreeressivee approach to minimize the time cost of DST models without negative impact on the model performance.
We adopt the concept of fertility proposed by Gu et al. (2018). Fertility denotes the number of times each input token is copied to form a sequence as the input to the decoder for non-autoregressive decoding. We first reconstruct dialogue state as a sequence of concatenated slot values. The result sequence contains the inherent structured representation in which we can apply the fertility concept. The structure is defined by the boundaries of individual slot values. These boundaries can be easily obtained from dialogue state itself by simply measuring number of the tokens of individual slots. Our model includes two-stage decoding process: (1) the first decoder learns relevant signals from the input dialogue history and generates fertility for each input slot representation; and (2) the predicted fertility is used to form a structured sequence which consists of multiple sub-sequences, each represented as (slot token slot fertility). The result sequence is used as input to the second decoder to generate all the tokens of the target dialogue state at once.
In addition to being non-autoregrresiive our models explicitly consider dependencies at both slot level and token level. Most of existing DST models assume independence among slots in dialogue states without explicitly considering potential signals across the slots (Wu et al.] 2019 Lee et al.]
2019 Goel et al. 2019 Gao et al. 2019). However, we hypothesize that it is not true in many cases. For example, a good DST model should detect the relation that train deearrrer should not have the (example in Table IT. Other cases include time-related pairs such as (taxi arriveer taxi lexi_le and cross-domain pairs such as (hotel_area, attraction_ars _area). Our proposed approach considers all possible signals across all domains and slots generate dialogue state as a set. Our approach directly optimizes towards the DST evaluation metrie Joint Accuracy (Henderson al. 2014b), which measures accuracy state (set of slots) level rather than slot level.
same value
Our contributions in this work include: (1) we propose a novel framework of -oo-eutorerressvve Dialog State Tracking (NADST), which explicitly learns inter-dependennies across slots for decod- ing dialogue states as a complete set rather than individual slots; (2) we propose a non-autoregressive decoding scheme, which not only enjoys low latency for real-time dialogues, but also allows to cap- ture dependencies at token level in addition to slot level; (3) we achieve the state-of-the oor-----rttttttt perfor- mance on the multi multi-domain task-triented dialogue dataset "MultiWOZ. 2.1" (Budzianowski et al., 2018 Fric et al., 2019) while significantly reducing the inference latency by an order of magnitude; (4) we conduct extensive ablation studies in which our analysis reveals that our models can detect potential signals across slots and dialogue domains to generate more correct "sets" of slots for DST.
RELATED WORK
Our work is related to two research areas: dialogue state tracking and non-autoregressive decoding.
2.1 DIALOGUE STATE TRACKING
Dialogue State Tracking (DST) is an important component in task- oriented dialogues, especially for dialogues with complex domains that require fine-grained tracking of relevant slots. Traditionally,
2

---

Published as a conference paper at ICLR 2020
DST is coupled with Natural Language Understanding (NLU). NLU output as tagged user utterances
is imput to DST models to update the dialogue states turn by turn (urrtaa et al. 2016 Shi et al. 2016;
Rastogi al. 2017). Recent approaches combine NLU and DST to reduce the credit assignment
problem and remove the need for NLU (Mrkšić et al., 2017 Xu & Hu 2018 Zhong et al. 2018).
Within this body of research, Goel et 2019) differentiates two DST approaches: fixed- and open- vocabulary. Fixed-vocabulary approaches are usually retrieval-
methods in which all candidate
pairs of (slot, value) from a given slot ontology are considered and the models predict a probability score for each pair (Henderson et al. 2014c] Ramadan et al. 2018 Lee et al. 2019). Recent work has moved towards open- vocabulayy approaches that can generate the candidates based on input text i.e. dialogue history (Lei et al., 2018 Gao et al. 2019; Wu et al.) (019). Our work is more related to these models, but different from most of the current work, we explicitly consider dependencies among slots and domains to decode dialogue state as a complete set.
2.2 NON-AUTOREGGRESSIVE DECODING
Most of prior work in non- or semi-autoregrrssive decoding methods are used for NMT to address the need for fast translation. [Shhwenk (2012) proposes estimate the translation model probabili- ties of a phase-based NMT system. Libovicky & Helel (018)) formulates the decoding process as a sequence labeling task by projecting source sequence into longer sequence and applying CTC loss (Graves et al. 2006) to decode the target sequence. [Wang et al. (019)) adds regularization terms to NAT models (Gu et al. 2018) to reduce translation errors such as repeated tokens and in- complete sentences. Ghazvininejad et al. (2019) uses a non-autoregressive decoder with masked attention to decode target sequences over multiple generation rounds. A common challenge non- autoregressive NMT is the large number of sequential latent variables, e.g., fertility sequences (Gu jet al., 2018) and projected target sequences (Libovický & Helel, 2018). These latent variables are used supporting signals for non- cemi-auutoeerrssivve decoding. We reformulate dialogue state as a structured sequence with sub-sequences defined as a concatenation of slot values. This form of dialogue state can be inferred easily from the dialogue state annotation itself whereas such super- vision information is not directly available in NMT. The lower semantic complexity of slot values as compared long sentences in NMT makes it easier adopt non-autoregressive approaches into DST. According to our review, we are the first to apply a non-autoregressive framework for generation- based DST. Our approach allows joint state tracking across slots, which results in better performance and an order of magnitude lower latency during inference.
3 APPROACH
Our NADST model is composed of three parts: encoders, fertility decoder. and state decoder, as shown in Figure [ The input includes the dialogue history X (II,..., EN) and a sequence of applicable (domain, slot) pairs Xas = ((d1 81), (do, $H)), where G and H are the total numbers of domains and slots, respectively. The output the corresponding dialogue states up to the current dialogue history. Conventionally, the output of dialogue state is denoted as tuple (slot, value) (or (domain-slot, value) for multi-domain dialogues). We reformulate the output as a concatenation of slot values yd:,;; Y = (Y ..... ,d) ) = (3) 3/2 ..... V1 y dr,ss
3/2 dr, ...... where I and J
the numbers of domains and slots in the output dialogue state, respectively.
First, the encoders use token-level embedding and positional encoding to encode the input dia- logue history and (domain, slor) pairs into continuous representations. The encoded domains and slots are then input to stacked self-attention and feed-forward network to obtain relevant signals across dialogue history and generate fertility ydo..s for cach (domain, slot) pair (dg, sh). The
output of fertility decoder is defined as sequence: Yfun ydi.ei ...., ...,,Yyla ydle where y Ydlacdh (0, (nxx(Sloee)ngth) max max(Slotl (SlotLength) = 9 according to the training data. We follow (Wu et al. 2019 Gao et al. 2019 to add slot gating mechanism as an auxiliary prediction. Each gate g is restricted to possible values: "none", "dontcare" and "generate"' They are used to form higher-level classification signals to sup- port fertility decoding process. The gate output defined as a sequence: Ygale Yyh ydo.en
For example, for the MultiWOZ dataset in our experiments, we have
g
The predicted fertilities are used to form an input sequence to the state decoder for non- autoregressive decoding. The sequence includes sub-sequences of (dg, 8h)) repeated by vdlu times
3

---

Published as a conference paper at ICLR 2020
and concatenated sequentially: ddsxfort = ((d1, 81) 81 ) ...., (dg. $ )r y and || d = || The decoder projects this sequence through attention layers with dialogue history. During this decoding process, we maintain a memory of hidden states of dialogue history. The output from the state decoder is used as a query to attend on this memory and copy tokens from the dialogue history to generate a dialogue state.
Following Lei al. (2018), we incorporate information from previous dialogue turns to predict current turn state by using partially delexicalized dialogue history Xeel (2i,del. .... NN,,,N as an input of the model. The dialogue history is delexicalized till the last system utterance by removing real-value tokens that match the previously decoded slot values to tokens expressed as domain-slot. Given a token Vn and the current dialogue turn t, the token is delexicalized as follows:
( (domainaa:-sotian if Tn C V- otherwise.
nel =deeex(x,) =
Inn,
domaingax = (icxx( slot,dx = Xdaxtent [id][[]]]]]]]]]]__]]]]]]]]] idx = Index( Y(-1) (2)
For example, the user utterance "I look for cheap hotel" is delexicalized to "I look for ho- tel pricerange hotel." if the slot hotel mootel_picerange is predicted as "cheap" in the previous turn. This approach makes use of the delexicalized form of dialogue history while not relying on an NLU module as we utilize the predicted state from DST model itself. In addition to the belief state, we also use the system action in the previous turn delexicalize the dialog history in a similar manner, following prior work (Rastogi et al. 2017 Zhong et al. 2018
Goel et al. 2019).
Fartiition
abi:
risat
masit
real
reat
Domain
Ro gatengon
Furtility
Gates
gen
gan
none
gen
kysw
mamrie
faod
price
Dielog History
Linear Softmax
Linear & Softmax
wont End restaurnt with priion food leoe for archiocture attracion near
Domain/Slot Token Embedding
Domain/Slot Positional Encoding
Token Embaciding
Partially Delex Context Attention
Domain Stot Token Self Self-Attention
Domainn-See Slot Pair Self Attention
Partialy Delex Cantext Attention
Positional Embedding
Context Attention
Domain/Siol Taken Embedding
find in restascurt arobitectue attracton near the centre
Linear Softmax
Pointer Network
Slots
Partilly Delesicalized Dising Hiltory
atr
aft
Stai
Saul
rest
centro
architect
collin
britsh
cheap
State
Generated
System
infomjies pritonshoop)
Dielog
previous
Dislog Hastory Delesicalization
Dosog Hattor
Portally Dolcaisalazdd
Chebag Hister
Our NADST has 3 key components: encoders ("red"), fertility decoder ("blue"), and state decoder ("green""). (i) Encoders encode sequences of dialogue history, delexicalized dialogue history, and domain and slot tokens into continuous representations; (ii) Fertility Decoder has attention mechanisms to learn potential dependencies across (domain, slot) pairs in combination with dialogue history. The output is used to generate fertilities and slot gates; and (iii) State Decoder receives the input sequence including sebssequences of (domain, slot)x fertilityy to decode a complete dialogue state sequence as concatenation of component slot values. For simplicity, we do not show feedforward, residual connection, and layer- normmalzaoiiiiioonnn layers in the figure. Best viewed in color.
Figure 1:
4

---

Published as a conference paper at ICLR 2020
3.1 ENCODERS
An encoder is used to embed dialogue history X into a sequence of continuous representations Z (21 RNxxd Similarly, partially delexicalized dialogue history X is encoded to contimuous representations Zdel E Nk We store the encoded dialogue history Z in memory which will be passed to a pointer network to copy words for dialogue state generation. This helps to address the OOV challenge as shown in (See et al. 2017 Wu et al.) 2019). We also encode each Te as input to the decoders. Each vector 2ds is used to store contextual signals for slot and fertility prediction during the decoding process.
(domain, slot) pair into continuous representation 2ds
Context Encoder. Context encoder includes a token-level trainable embedding layer and layer normalization (Ba et al.6]2222111 The encoder also includes a positional encoding layer which follows sine and cosine functions (Vaswani et al., 2017). An element-wis summation is used to combine the token-level vectors with positional encoded vectors. We share the embedding weights to embed the raw and delexicalized dialogue history. The embedding weights are also shared to encode input to both fertility decoder and state decoder. The final embedding of X and Xdet is defined as:
Z Zemb + PE(X) E IR Nxd
Zdel Zemb,del + PE((del) E IR Nxd
(4)
Domain and Slot Encoder. Each (domain, slot) pair is encoded by using two separate embedding vectors of the corresponding domain and slot. Each domain g and slot is embedded into a contin- uous representation Fd, and žan, E IRd The final vector is combined by element-wise summation:
(5) We share the embedding weights to embed domain and slot tokens in both fertility decoder and state decoder. However, for input to state decoder, we inject sequential information into the input Xdsxfert to factor in position -iite information to decode target state sequence. In summary, Xas and daxttert is encoded as following:
#d = 2dg + 2n E Rd
Zemb.da 2di,01 ① ... 2d Zasxfort = dembe + PPPPP((aaa PF((Xdsxfort)
Zaa=
(6)
(7)
yo
dex = (ad)
4 @ (Ede.e))
(8)
where + denotes concatenation operation. Note that different from a typical decoder input in Transformer, we do not shift the input sequences to both fertility decoder and state decoder by one position as we consider non-autoregressive decoding process in both modules. Therefore, all output tokens are generated in position based on all remaining positions of the sequence i.e.
1,..., i - 1, X,aa || in fertility decoder and 1,..., i 1--1, 1,, 1, 1,...|||Xaxxtte in state decoder.
3.2 FERTILITY DECODER
Given the encoded dialogue history Z, delexicalized dialogue history Zaet, and (domain,slot) pairs Zas, the contextual signals are learned and passed into each 2ds vector through a sequence of at- tention layers. We adopt the multi-head attention mechanism (Vaswani et al., 2017) to project the representations into multiple sub-spaces. The attention mechanism is defined as scaled dot-product attention between query Q key K, and value V:
Attention(Q. K, V) = softmax( QKT VoL V)
Each multi-head attention is followed by position-wise forward network. The feed-forward is applicd to each position separately and identically. We use two linear layers with a ReLU activa- tion in between. The fertility decoder consists of attention layers, each of which lears relevant contextual signals and incorporates them into 2ds vectors as input to the next attention layer:
Zoul
Attention( Zds, Zels, 7dx) E R RxN
(10)
Zou
Attention Zoout Zdel, Zael) E Rxd
(11)
Zout
Attention Zout Z, Z) E R Nxd
(12)
5

---

Published as a conference paper at ICLR 2020
For simplicity, we do not express the multi-head and Seed-forward equations. We advise the reader to review Transformer network (Vaswani et al. 2017) for more detailed description. The multi-head structure has shown obtain good performance in many NLP tasks such as NMT (Vaswani et al. 2017) and QA (Dehghani et al. 2019). By adopting this attention mechanism, we allow the models explicitly obtain signals of potential dependencies across (domain, slot) pairs in the first attention layer, and contextual dependencies in the subsequent attention layers. Adding the delexicalized dia- logue history input can provide important contextual signals as the models can learn the mapping between real-value tokens and generalized domain-slor tokens. To further improve the model capa- bility capture these dependencies, we repeat the attention sequence for Tiert times with d In an attention step I, the output from the previous attention layer | 1 is used as input to current layer to compute Zi.- The output in the last attention layer Tius is passed to two independent linear transformations predict fertilities and gates:
pgate softmax W "gaate late4darr [iurs)
(13)
pfert softmax( W ZTme)t
(14)
We use the standard cross-entropy loss to train the where W gate E Rdx3 and Wiert E pdx10 prediction of gates and fertilities:
Cgate
-log( (Ydg.)))
(15)
dyrsh
Liort
-log -log (pfort ( (Ygloven)
(16)
dg.sh
3.3 STATE DECODER
Given the generated gates and fertilities, we form the input sequence Xdsxfert- We filter out any (domain, slot) pairs that have gate either as "none" or "dontcare". Given the encoded input Zasx fert> we apply a similar attention sequence used in the fertility decoder to incorporate contextual sig- nals into each zdsxfert vector. The dependencies are captured the token level this decoding stage rather than at domain/slot higher level as in the fertility decoder. After repeating the attention sequence for stat times, the final output ds fort is used to predict the state in the following:
vocab = softmax(W (Sattt 2 Ziatatn
(17)
where W state E RdxIIVII with V as the set of output vocabulary. As open-vocabulary DST models do not assume a known slot ontology, our models can generate the candidates from the dialogue history itself. To address OOV problem during inference, we incorporate a pointer network (Vinyals et al. 2015) into the Transformer decoder. We apply dot-produce attention between the state decoder output and the stored memory of encoded dialogue history Z:
pstate softmax TTesaee Z²)
(18)
The final probability of predicted state is defined as the weighted sum of the two probabilities:
petate state vocab + (1 -ptatte pstate
(19)
ptr
Pgtate = sigmoid Wyenn Vgen)
(20)
V yen
Z fort
ZTitttt dasiterr Zexp
(21)
where Wgen E 3dx and Zeep expanded vector of Z to match dimensions Zas final probability is used to train the state generation following the cross-entropy loss function:
The
ydgi
Estate
Σ
Log(pstate (y (1
pstate dg.
(22)
dg.58
m==
3.4 OPTIMIZATION
We optimize all parameters by jointly training to minimize the weighted sum of the three losses:
L
Catate + a&gatt + 3 3
(23)
where > 0 and are hyper-parameters.

---

Published as a conference paper at ICLR 2020
4 EXPERIMENTS
4.1 DATASET
MultiWOZ (Budzianowski al. 2018 is one of the largest publicly available multi-domain task- oriented dialogue dataset with dialogue domains extended over 7 domains. In this paper, we use the new version of the MultiWOZ dataset published by EEric et al. (019). The new version includes some correction dialogue state annotation with more than 40% change across dialogue turns. On average, each dialogue has more than one domain. We pre-processed the dialogues by tokenizing. lower-casing, and delexicalizing all system responses following the pre-processing scripts from (Wu et al. 2019p. We identify a total of 35 (domain, slot) pairs. Other details of data pre-processing procedures, corpus statistics, and list of (domain, slot) pairs are described in Appendix A.1
4.2 TRAINING PROCEDURE
We use label smoothing (Szegedy et al., 2016) to train the prediction of dialogue state Y but not for prediction of fertilities fert and gates Y gate- During training, we adopt 100% teacher-forcing learning strategy by using the bround-truth of Xdsxfent as input to the state decoder. We also apply the same strategy obtain delexicalized dialogue history i.c. dialogue history delexicalized from the ground-truth truth belief state in previous dialogue turn rather than relying on the predicted belief state. During inference, we follow a similar strategy as (Lei al.) 2018 by generating dialogue state turn-by-turn and use the predicted belief state in turn 1 to delexicalize dialogue history in turn t. During inference, Xds fert is also constructed by prediction Ygate and Yiert- We adopt the Adam optimizer (Kingma Ba 2015) and the learning rate strategy similarly as (Vaswani et al., 2017). Best models are selected based the best average joint accuracy of dialogue state prediction in the validation set. All parameters are randomly initialized with uniform distribution (Glorot & Bengio 2010). We did not utilize any pretrained word- or character- based embedding weights. We tuned the hyper-parameters with grid-search over the validation set (Refer to Appendix [A.22 for further details). We implemented our models using PyTorch (Paszke et al. 2017) and released the code on GitHub
4.3 BASELINES
The DST baselines can be divided into 2 groups: open-vocabulary approach and fixed-vocabulary approach as mentioned in Section Fixed-vocabllary has the advantage of access to the known can- didate set of each slot and has high performance of prediction within this candidate set. However, during inference, the approach suffers from unseen slot values for slots with evolving candidates such as entity names and time- and location-related slots.
4.3. FIXED-VOCABULARY
GLAD (Zhogg et al., 2018). GLAD uses multiple self-attentive RNNs to learn a global tracker for shared parameters among slots and a local tracker for individual slot. The model utilizes previous system actions as input. The output is used to compute semantic similarity with ontology terms.
GCE (Nouri & Hosseini-Asl] 2018). GCE is simplified and faster version of GLAD, The model removes slot-specific RNNs while maintaining competitive DST performance.
MDBT (Ramadan et al. 2018). MDBT model includes separate encoding modules for system utterances, user utterances, and (slot, value) pairs. Similar to GLAD, The model trained based on the semantic similarity between utterances and ontology terms.
FJST and HJST (Eric et al. 2019). FIST refers to Flat Joint State Tracker, which consists a dialog history encoder as a bidirectional LSTM network. The model also includes separate feedforward networks to encode hidden states of individual state slots. HIST follows a similar architecture but uses hierarchical LSTTM network (Serban al. 2016) to encode the dialogue history.
SUMBT (Lee et al. 2019). SUMBT refers to Slot-independent Belief Tracker, consisting of a multi-head attention layer with query vector a representation of (domain, slot) pair and key and
httpp:://github.comm/eeee
7

---

Published as a conference paper at ICLR 2020
value vector BERT-encoded dialogue history. The model follows non-parametric approach as it is trained to minimize a score such as Euclidean distance between predicted and target slots. Our approach is different from SUMBT learn dependencies among the pairs. Our models also generate slot values rather than relying on a fixed candidate set.
include attention among (domain, slot) pairs to explicitly
4.3.2 VOCALLYCAR
TSCP Lei ct al. 2018). TSCP is an end-to-end dialogue model consisting of an RNN encoder and two RNN decoder with a pointer network. We choose this as baseline because TSCP decodes dialogue state as single sequence and hence, factor in potential dependencies among slots like our work. We adapt TSCP into multi-domain dialogues and report the performance of only the DST component rather than the end-to-end model. We also reported the performance of TSCP for two cases when the maximum length of dialogue state sequence L in the state decoder is set to 8 or 20 tokens. Different from TSCP, our models dynamically learn the length of each state sequence the sum of predicted fertilities and hence, do not rely on a fixed value of L.
DST Reader (Gao et al. 2019). DST Reader reformulates the DST task a reading comprehension task. The prediction of each slot is span over tokens within the dialogue history. The model follows attention-ban based neural network architecture and combines slot carryover prediction module and slot type prediction module.
HyST (Goel et al. 2019). HyST model combines both fixed-vocabulary and open-vocabulary approach by separately choosing which approach is more suitable for each slot. For the open- vocabulary approach, the slot candidates formed sets of all word n-grams in the dialogue history. The model makes use of encoder modules to encode user utterances and dialogue acts to represent the dialogue context.
TRADE (Wu et al. 2019). This is the current state-of-the-art model on the MultWWOZ2.0 and 2.1 datasets. TRADE composed of a dialog history encoder, a slot gating module, and an RNN decoder with a pointer network for state generation. SpanPtr is a related baseline to TRADE as reported by Wu al. (219)). The model makes use pointer network with index-based copying instead of token-based copying mechanism.
4.4 RESULTS
We evaluate model performance by the joint goal accuracy as commonly used in DST (Henderson) et al.) 2014b). The metric compares the predicted dialogue states to the ground truth in each di- alogue turn. A prediction only correct all the predicted values all slots exactly match the corresponding ground truth labels. We ran models for 5 times and reported the average results. For completion, we reported the results in both MultiWOZ 2.0 and 2.1.
As can be seen in Table although models designed for non non-autoregressive decoding, they can outperform state-of-the-art DST approaches that utilize autoregressive decoding such as (Wul al., 2019). Our performance gain can be attributed the model capability of learning cross- domain and cross-slot signals, directly optimizing towards the evaluation metric of joint goal accu- racy rather than just the accuracy of individual slots. Following prior DST work, we reported the model performance on the restaurant domain in MultiWOZ 2.0 in Table In this dialogue domain, our model surpasses other DST models in both Joint Accuracy and Slot Accuracy. Refer to Ap- pendix A.3 for our model performance in other domains in both MultiWOZ2.r and MultWWOZ.1.
Latency Analysis. We reported the latency results term of wall-clock time (in ms) per prediction state of our models and the two baselines TRADE (Wu et al. 2019) and TSCP (Lei et al. 2018) in Table 4 For TSCP, we reported the time cost only for the DST component instead of the end-to- end models. We conducted experiments with 2 cases of TSCP when the maximum output length of dialogue state sequence in the state decoder is set as L = and L = 20. We varied our models for different values 7' Tiert Tstate E {1, 2, 3). All latency results are reported when running in single identical GPU. As can be seen in Table NADST obtains the best performance when 3. The model outperforms the baselines while taking much less time during inference. Our approach similar to TSCP which also decodes complete dialogue state sequence rather than individual slots to factor in dependencies among slot values. However, as TSCP models involve sequential
8

---

Published as a conference paper at ICLR 2020
Model
MultiWOZ2.1
MultiWOZ2.00
MDBT
(Ramadan et al.
2018
13.57%
SpanPtr Vinyals et al.
2015)
30.28%
GLAD
hongg et al.
2018
35.57%
GCE Nouri & Hosseini-Ar
2018
36.27%
HJST
Erc et al. 2019)
35.55%
38.40%
DST Reader (single) (Gao et al. 2019)
36.40%
39.41%
DST Reader (ensemble)
Gao et al. 2019
42.12%
TSCP (Lei al.
2018
37.12%
39.24%
FJST (Eric et al.
2019
38.00%
40.20%
HyST (ensemble)
(Goel et al. 2019
38. 10%
44.24%
SUMBT (Lee et al. 2019
46.65%
TRADE Wu ct al.
2019)
45.60%
48.60%
Ours
49.04%
50.52%
Table DST Joint Accuracy metric on MultWWOZ 2.1 and 2.0. t; results reported on MultiWOZ2.0
leaderboard. *: results reported by Fric et al. (2019). Best results are highlighted in bold.
processing in both encoding and decoding, they require much higher latency. TRADE shortens the latency by separating the decoding process among (domain, slo() pairs. However, at the token level, TRADE models follow an auto-regressive process to decode individual slots and hence, result in higher average latency compared
approach. In NADST, the model latency is only affected by the number of attention layers in fertility decoder Tiert and state decoder Tsaate For approaches with sequential encoding and/or decoding such as TSCP and TRADE, the latency is affected by the length of source sequences (dialog history) and target sequence (dialog state). Refer to Appendix A.3 for visualization of model latency in terms of dialogue history length.
Model
Joint Acc
Slot Ace
Model
Joint Acc Latency Speed Up
MDBT
17.98%
54.99%
TRADE
45.60%
362.15
x2.12
SPanPtr
49.12%
87.89%
TSCP (L=8)
32.15%
493.44
x 1.56
GLAD
53.23%
96.54%
TSCP (I=20)
37.12%
767.57
1.00
GCE
60.93%
95.85%
Ours (T=1)
42.98%
15.18
x50.56
TSCP
62.01 %
97.32%
TRADE 65.35%
93.28%
Ours (T=2)
45.78%
21.67
x35.42
Ours 69.21
98.84%
Ours (T=3)
49.04%
27.31
x28.11
Table DST joint accuracy and slot accuracy MultWWOZ2.0 restaurant domain. Baseline re- sults (except TSCP) were from Wu al. (2019).
Table 4: Latency analysis on MultiWOZ2.1. Latency is reported in terms of wall-clock time in ms per prediction state.
Ablation Analysis. We conduct extensive ablation analysis with several variants of mod- els Table [ Besides the results of DST metrics, Joint Slot Accuracy and Slot Accuracy, we reported the performance of the fertility decoder in Joint Gate Accuracy and Joint Fertility Accu- racy, These metrics are computed similarly as Joint Slot Accuracy in which the metrics are based on whether all predictions of gates fertilities match the corresponding ground truth labels. We also reported the Oracle Joint Slot Accuracy and Slot Accuracy when the models are fed with ground truth Xdax fert and Xdel labels instead of the model predictions. We noted that the model fails when positional encoding of Xdsxfort. is removed before being passed to the state decoder. The perfor- mance drop can be explained because PE responsible for injecting sequential attributes to enable non-autoregressive decoding. Second, we also note a slight drop of performance when slot gating is removed as the models have to learn to predict a fertility of for "none" and "dontcare" slots as well. Third, removing Xael in Joint Fertility Accuracy. Lastly, removing pointer generation and relying on only Petate affects the model performance the models not able to infer slot values unseen during training, espe- cially for slots such as restaurant-nane and arain-arriveb.. We conduct other ablation experiments and report additional results in Appendix A.3
input reduces the model performance, mostly due to the sharp decrease

---

Published as a conference paper at ICLR 2020
Slot Gating
Pointer Gen.
Joint Gate
Joint Fert.
Joint
Slot
Oracle Joint
Oracle
Acc
Acc
Slot Acc
Acc
Slot Acc
Slot Acc
66.65 %
63.18%
49,04%
97.31%
73.44%
99.01 %
59. 39.23%
57.83%
19.56%
94.36%
72. 12%
98.96%
N/A
64.23%
48.74%
96.62%
73.01%
98.97%
48. 48.23%
45,35%
39.45%
95 92%
66.27%
98.63%
((no sys, act)
52.45%
56.81%
44.87%
70.83%
98.74%
63.19%
58.31%
43.46%
96.72%
64.37%
98,39%
44.22%
42.01%
N/A
41.35%
33,52%
95.42%
60.99%
98.19%
Table 5: Ablation analysis on MultiWOZ 2.1 4 components: partially delexicalized dialogue history Xdet, slot gating, positional encoding PE PE(Xaexfert) and pointer network.
Auto-regressive DST. We conduct experiments that use an auto-regressive state decoder and keep other parts of the model the same. For the fertility decoder, do not use Equation [4 and 16 as fertility becomes redundant in this case. We still use the output to predict slot gates. Similat lar to TRADE, use the summation of embedding vectors of each domain and slot pair input to the state decoder and generate slot value token by token. First, From Table 6 we note that the perfor- mance does not change significantly compared to the non-autoregressivee version. This reveals that our proposed NADST models can predict fertilities reasonably well and performance is com- parable with the rugo-segressive approach. Second, we observe that the auto-regressive models are less sensitive to the use of system action in dialogue history delexicalization. We expect this as pre- dicting slot gates is easier than predicting fertilities. Finally, we note that our auto-regressive model variants still outperform the existing approaches. This could be due to the high-level dependencies among (domain, slot) pairs learned during the first part of the model to predict slot gates.
Joint Gate
Joint
Slot Acc
Oracle Joint
Oracle Slot Acc
MultiWOZ
Sys. Act
Acc
Slot Ace
Slot Acc
2.1
65,89%
49.76%
97.40%
71.39%
98.92%
2.1
62.04%
46.57%
97.23%
66. 72%
98.65%
2.0
68,81 68.81%
50.08%
97.44%
79.04%
99.22%
2.0
65.27%
50.46%
97.43%
76.21% %
99.08%
Table 6: Performance of auto-regressive model variants on MultiWOZ2.0 and 2.1. Fertility predic- tion is removed fertility becomes redudant in auto-regressive models.
Visualization and Qualitative Evaluation. In Figure we include two examples of dialogue state prediction and the corresponding visualization of self-attention scores of X fert in state decoder. In each heatmap, the highlighted boxes express attention scores among non-symmetrical domain-slot pairs. In the first row, attention heads capture the dependencies of two pairs (train- leaveat, rrai--arrveby) and (train-departure, vrain-destination). The model prediction for these two slots matches the gold labels: cambridge). fraai-destination. ely) respectively. In the second row, besides slot-level dependency between domain-slot pairs (laxi-departure, taxi-destination) token-level dependency is exhibited through the attention between aftraction-type and attraction-name. By attending token rep- resentations of attraction-name with corresponding output "christ college", the models can infer cuttraction-tpp=coolleee correctly. In addition, our model also detects contextual dependency be- tween train-de and aftraction-nane to predict ftrain-depatuureehhee college." Refer to Ap- pendix A.4 for the dialogue history with gold and prediction states of these two sample dialogues.
09:50), ttrain-arrivdd
11:30) and (train- -rearurd
CONCLUSION
We proposed NADST, novel -on-Autoreeresssvv neural architecture for DST that allows the model to explicitly learn dependencies at both slot-level and token-level to improve the joint accuracy rather than just individual slot accuracy. Our approach also enables fast decoding of dialogue states by adopting parallel decoding strategy in decoding components. Our extensive experiments on the well-known MultiWOZ corpus for large-scale multi-domain dialogue systems benchmark show that our NADST model achieved the state-of-the-art accuracy results for DST tasks, while enjoying substantially low inference latency which is an order of magnitude lower than the prior work.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint ar0iv: 1607.004006 2016.
Pawel Budzianowski, Tsung- Tsung-Hsien Hsien Wen, Bo-Hsiang Tseng, lūigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. MultiWOZ a large-scale multi-domain wizard-of-Oz dataset for task-oriented dialogue modelling. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016-5026, Brussels, Belgium, October-November 2018 Association for Computational Linguistics. doi: 10.18653/1//D1--- 1547. URL https:://ww.
aclweb.org/ http:/orrg..nnooooooooeee.e. 1547
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal
transformers. In International Conference on Learning Representations, 2019. URL https:
//openzeview.co net. fmr wttprrmmmmmddeydderr1222
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur, Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv:1907.000 0066966,,99 2019.
Shuyang Gao, Abhishek Sethi, Sanchit Aggarwal, Tagyoung Chung, and Dilek Hakkani-Tuu Dialog state tracking: A neural reading comprehension approach. arXiv preprint arXiv: 1908. 01946, 2019.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Constant-time machine translation with conditional masked language models. arXiv preprint arXiv:1904.099 1904.09324, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Rahul Goel, Shachi Paul, and Dilek Hakkani-Tür. Hyst: A hybrid approach for flexible and accurate dialogue state tracking. Proc. Interspeech 2019, pp. 1458-1462, 2019.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Pro- ceedings of the 23rd international conference on Machine learning, pp. 369-376. ACM, 2006.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive
neural machine translation. International Conference on Learning Representations, 2018. URL, https: /ittps:/copenrest ew net ww..ee/oorummmmmmmt///// Porrumiidddp (Bt Cb
Matthew Henderson, Blaise Thomson, and Jason D Williams. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp. 263-272, 2014a.
Matthew Henderson, Blaise Thomson, and Steve Young. Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp. 292-299, 2014b.
Matthew Henderson, Blaise Thomson, and Steve Young. Robust dialog state tracking using delex- icalised recurrent neural networks and unsupervised adaptation. 2014 IEEE Spoken Language Technology Workshop (SLT). pp. 360-365, 2014c.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Con- ference on Machine Learning, pp. 2395-2404, 2018
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
11

---

Published as a conference paper at ICLR 2020
Gakuto Kurata, Bing Xiang, Bowen Zhou, and Mo Yu. Leveraging sentence-level information with encoder LSTM for semantic slot filling. In Proceedings of the 2016 Conference on Em- pirical Methods in Natural Language Processing, pp. 2077-2083, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 10.18653/11/D1-- 1223. URL https:: 7 /ww. ac www.wlee.bcoo www.ee/..rgggatooooooo//iiieeee.
Hwaran Lee, Jinsik Lee, and Tae yoon Kim. Sumbt: Slot-utterance matching for universal and scalable belief tracking. In ACL, 2019.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural se- quence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1173-1182, 2018.
Wenqiang Lei, Xisen Jin, Min- Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. Sequicity: Simplifying task dialogue systems with single sequence- seutencett--eqqeeee to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1437-1447, 2018.
Jindrich Libovický and Jindřich Helcl. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3016-3021, 2018
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp. 1412-1421, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/1//D15-1116 URL www.wwww.... acl web.
attg/anthollog/Dil5-11
Nikola Mrkšić, Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1777- 1788. Association for Computational Linguistics, 2017. doi: 10.18653/w1/11--1163. URL http://wwe ac www.wwwwwwbbb..........
Elnaz Nouri and Ehsan Hosseini-Asl. Toward scalable neural dialogue state tracking model. arXiv preprint arXiv:1812.00889, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Osman Ramadan, Pawet Budzianowski, and Milica Gasic. Large-scale multi-domain belief track- ing with knowledge sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume 2, pp. 432 437, 2018
Abhinav Rastogi, Dilck Z. Hakkani-Türe and Larry P. Heck. Scalable multi-domain dialogue state tracking. 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 561-568, 2017.
Holger Schwenk. Continuous space translation models for parase-based statistical machine trans- lation. In Proceedings of COLING 2012: Posters, pp. 1071-1080, Mumbai, India, Decem- ber 2012. The COLING 2012 Organizing Committee. URL https: https::/www..ccl. aclweb. www.eee.rrr/.....
anthology/C12-2115
Abigail See, Peter J Liu, and Christopher D Manning. Get the point: Summarization with pointer- generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Pupers), pp. 1073-1083, 2017.
Julian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference Artificial Intelligence, 2016.
12

---

Published as a conference paper at ICLR 2020
Yangyang Shi, Kaisheng Yao, Hu Chen, Dong Yu, Yi-Cheng Pan, and Mei- Yuh Hwang. Re- current support vector machines for slot tagging in spoken language understanding. In Pro- ceedings of the 2016 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies, pp. 393-399, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18633/11/N16-1000
URL
1044
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting The Journal of Machine Learning Research, 15(1):1929-199 1958, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2822 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998-60008 Curran Associates, Inc., 2017. URL http::/papers: nips.npppn..
wttp:////t--aattteetoo---------- -ol---ooo-----l-------eeeddddd pdf
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Gareett (eds.), Advances in Neural Information Processing Systems 28, pp. 2692-2700. Curran Associates, Inc., 2015. URL httt:::/ppaers..ttt nips.
cC/ htc/paper/55 58 8- nte.--etwerrreee.......mm
Yiren Wang, Fei Tian, Di He, Tao Qin. ChengXiang Zhai, and Tie-Yan Liu. Non-autoregreesive ma- chine translation with auxiliary regularization. In The Thirty- Third AAAI Conference on Artificial Intelligence (AAAI2019), pp. 5377-5384, 2019.
''suggHHieen Wen, David Vandyke, Nikola Mrkšić, Milica Gasic, Lina M. Rojas Barahona, Pei- Hao Su, Stefan Ultes, and Steve Young. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Volume 1, Long Papers, pp. 438-449, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https: www. ac www...ew.b.... oebgg anthology/EI7-104
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosscini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 808 819, Florence, Italy, July 2019. Association for Computational Linguistics. URL
https:/ / http://wwllll....... wttp://ww..bb.oor////.oo.ooo///.
Puyang Xu and Qi Hu. An end-to-end approach for handling unknown slot values in dialogue state tracking. In Proceedings of the 56th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1448-14577 Association for Computational Linguistics, 2018. URL www.wb.wwwbbb.....cc............ org wwtp/artoooogo///eeee.e.o
Victor Zhong, Caiming Xiong. and Richard Socher. Global-locally self-attentive encoder for di- alogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1458- 1467, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.186633v1//18. 1135. URL https:
7/www. ac www.cleebbooo www.:aawwwb..orggaaaatloooo//o/.
13

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 DATASET PRE-PROCESSING
We follow similar data preprocessing procedures as Budziannowkki on both MultiWZ 2.0 and 2.1. The resulting corpus includes 8,438 multi-turn dialogues in training set with an average of 13.5 turns per dialogue. For the test and validation set, each includes 1 1,000 multi-turn dialogues with an average of 14.7 turns per dialogue. The average number of domains per dialogue 1.8 for training, validation, and test sets. The MultiWOZ corpus includes much larger ontology than previous DST datasets such as WOZ (Wen et al. (017)) and DSTC2 (Henderson et al.) 2014ap. We identified a total 35 (domain, slot) pairs across domains. However, only domains included in the test data. Refer to Table for the statisties of dialogues in these domains.
(2018) and Wu et al. (2019)
Domain
aftraction
hotel
restaurant
taxi
train
All
bookday
bookpeople
area bookday
arriveby bookpeople day departure destination
bookstay
arriveby departure destination leaveat
area name
internet
bookpeople booktime food
Slot
name
type
parking
pricerange
leaveat
pricerange
stars
type
frain
3,381
3,103
2,717
3,813
1,654
8,438
val
416
484
401
438
207
1,000
test
394
494
395
437
195
1,000
Table 7: Summary of MultiWOZ dataset 2.1
A.2 MODEL SYYPERPPARAMETE
We employed dropout (Srivastava et al. 2014 of 0.2 at all network layers except the linear layers of generation network components and pointer attention components. We used batch size of 32, embedding dimension = 256 in all experiments. We also fixed the number of attention heads to 16 all attention layers. We shared the embedding weights to embed domain and slot tokens input to fertility decoder and state decoder. We also shared the embedding weights between dialogue history encoder and state generator. We varied our models for different values T Tertt Tstate (1, 3). In all experiments, the warmup steps are fine-tuned from range from 13K to 20K training steps.
A.3 ADDITIONAL RESULTS
Domain-specifie Results. We conduct experiments evaluate model performance in all 5 test domains MultiWO72.0 and 2.1. From Table our models perform better in restaurant and at- traction domain in general. The performance in the taxi and hotel domain is significantly lower than other domains. This could be explained as the hotel domain has a complicated slot ontology with 10 different slots, larger than the other domains. For the taxi domain, this domain are usually of multiple domains, including the taxi domain in combination with other domains. Hence, it more challenging to track dialogue states in the taxi domain.
observed that dialogues with
Latency Results. We visualized the model latency against the length of dialogue history in Figure 2 and In Figure 2 we only plot with dialogue history length up to 80 tokens as TSCP models do not use the full dialogue history as input. In Figure 3 for fair comparison between TRADE and NADST, we plot the latency of the original TRADE which decodes dialogue state slot by slot and new version of TRADE" model which decodes individual slots following parallel decoding mechanism. Since TRADE independently generates dialogue state slot by slot, we enable parallel generation simply by feeding all slots into models at (without impacts performance). How- ever, at the token level, TRADE* still follows an autoregressive decoding framework. Compared to TRADEF and TSCP, our model latency is only dependent the model complexity i.e. the number
14

---

Published as a conference paper at ICLR 2020
MultiWOZ2.1
MultiWOZ2.0
Domain
Joint Ace
Slot
Joint Acc
Slot
Hotel
48.76%
97.70%
53.86%
97.75%
Train
62.36%
98.36%
58.58%
98.08%
Attraction
66.83%
98.89%
74.21%
99.19%
Restaurant
65.37%
98.78%
69.21
8..8%%
Taxi
33.80%
96.69%
34.94%
96.76%
Table 8: Additional domain-specific results of our model in MultiWOZ2.0 and MultWWO2.11 The model performs best with the restaurant domain and worst with the taxi domain.
of attention layers T Tertt T Tetate- For TRADE" and TSCP, the model latency increases as dialogue extends over time while NADST latency is almost constant. The non-constant latency is mostly due to overhead processing such as delexicalizing dialogue history. Our approach is, hence, suitable especially for dialogues in multiple domains as they usually extend over more number of 14 turns per dialogue in average MultiWOZ corpus) In Figure 3] we noted that the latency of the original TRADE is almost unchanged as the dialogue history extends. This is most likely due to the model having to decode all possible (domain, slot) pairs rather than just relevant pairs as in NADST and TSCP. The TRADE shows clearer increasing trend of latency because the parallel process independent the number of (domain.slot) pairs considered. TRADE* still requires more time decode than NADST also parallelize decoding the token level.
turns (e.g.
Ours(T=1)
1200
Ours(T=2)
Ours(T=3)
TSCP(L=8)
latency (ms/predicted state)
1000
TSCP(L=20) 20)
800
600
400
200
0
10
20
30
40
50
60
70
80
dialog history length (tokens)
Figure 2: Comparison of model latency wall-clock time (in ms) per prediction of complete dia- logue state (not by individual slot). The latency plotted against the length of the dialogue history. We compare our models with TSCP (Lei et al.) 2018) with varied maximum output length of dia- logue states L = and L 20. We vary our models with different values number of attention layers T Tyert Tatate 2, 3. Our models more scalable the latency does not change significantly when dialogue history extends over time.
Ablation Results. We conduct additional ablation experiments by varying the proportion of predic- tion values ground-truth values for X and Xeex input to the models. As be in Table 9 the model performance increases gradually as the proportion of prediction input %pred reduces from 100% (true prediction) to 0% (oracle prediction). In particular, observe more sig-
15

---

Published as a conference paper at ICLR 2020
base-10 logarithm latency ims/predicted state)
Our)(
2.6
Ours(T=2)
Ours(T=3)
2.4
TRADE
TRADE*
2.0
1.8
1.6
1.2
0
100
200
300
400
500
dialog history length (tokens)
Figure 3: Comparison of model latency as wall-clock time (in ms) per prediction of complete dia- logue state. The latency plotted against the length of the dialogue history. For fair comparison, we compare our models with TRADE (Wu et al. 2019) in 2 cases: original TRADE which decodes dialogue state slot by slot and TRADE* which decodes dialogue state parallel at slot-level. Here plot the base-10 10 logarithm of latency to show the difference between the 2 cases of TRADE. We vary our models with different values of number of attention layers T Tyert = Tatate 1, 2. Our models more scalable the latency does not change significantly when dialogue history extends over time.
nificant changes in performance against changes of %pred of Xdex The model performance increase up to more than 67% joint accuracy when we have an oracle input of Xdex fert- However, consider improving model performance by X more practically achievable. For example, can make use of a more sophisticated mechanism to delexicalize dialog history rather than exact word matching as the current strategy. Another example is having better Xast through pretrained NLU model. In the ideal case with access to ground-truth labels of both Xáet and Xdax fert. the model obtain a joint accuracy of 73%.
%pred
%
Joint Acc
Slot
% Poped
% Veeedd
Joint
Slot
Xaat
Xasx
Ace
X
X fert
Acc
Acc
0%
100%
57.40%
98.06%
100%
0%
67.32%
98.67%
20 %
100 %
56.50%
97.98%
100%
20%
64.09%
98.47%
40 %
100 %
55.24%
97.91% %
100%
40%
61.29%
98.29%
60 %
100 %
53.58%
97.79%
100% %
60%
57.02%
98.00%
80 %
100 %
52.02%
97.67%
100%
80%
54.11%
97.76%
100%
100%
49.04%
97.31%
100%
100 %
49.04%
97.31%
0 %
0%
73.44%
99.01 %
0 0%
0% %
73.44%
99.01% %
Table 9: Additional results of our model in MultiWOZ2.1 when we assume access to the ground- truth labels of Xact and X fert (oracle prediction). We vary the the percentage of using the model prediction Xast and Xasx fert from 100% (true prediction) to 0% (oracle prediction).
16

---

Published as a conference paper at ICLR 2020
attoctionn name
atbrction reme
post
ard
atroction
Falramoeey
o 11200
can-departur
Eet.dentination
D6:30
Figure 4: Heatmap visualization of self- scores of 5 heads between Zdsx fert representa- tions in the state decoder. The corresponding prediction output for each representation is presented on the right side. The examples are for the 6th turn in dialogue ID MUL00336 (upper row) and PMUL3759 (lower row) in MultiWO72.1.
A.4 SAMPLE PREDICTION OUTPUT
We extracted prediction output in all turns for 2 example dialogues: MUL0536 and PMUL3759.
17

---

Published as a conference paper at ICLR 2020
Human:
am looking for abbey pool astroturf pitch can you help me?
Turn
Gold Dialog State:
(atraaciionname, abley pool and astrocurf pitch)
Predicted Dialog State:
(altraction-name, abbey pool and astrocurf pitch)
yes, abbey pool and astroturf pitch swimmingpool cast of town. their number 01 01223902088, and address is pool way. whitehill mad, off newmarket road. posicode ch5ätr. thank you very
System:
Turn 2
Human:
the information. that all accded help with. nice day. (atraaction-area cast), (attracion-name, abbey pool and astroturf pitch), (attraction-tyye, swimming pool) (ottraction-aren. east), (atractioe-name, abbey pool and astroturf pitch)
Gold Dialog State:
Predicted Dialog State:
System:
you are weleome. ler me know if can do anything else for you.
Human:
actually do need a train going to cly-
Turn 3
(utrraction-aren, east), nntracttionname abbey pool and astroturi pitch), (attraction-tyye, swimming pool).
Gold Dialog State:
(train (train-destinan destination-ely)
Predicted Dialog State:
(attraction-ara, cast) (atti antioonmmm, abboy pool and astroturf pitch). (train-dstiiatiooeel))
System:
that leaving from cambridge ? and so, what time would you like to arrive in ely?
Human:
yes, will be leaving cambridge and going to cly would like it anrve by 11:30.
Turn 4
(ottraction-area cast), (attractionnname, abbey pool and astroturf pitch), (attraction-lype, swimming pool),
Gold Dialog State:
(train anriveyy. 11:30), (train departure, cambridge). (train- destination, ely)
(uttractien-area, area, cast), (attraction-nann) name, abbey pool and astroturf pitch). (train aariveyrrveby..... 11:30).
Predicted Dialog State:
(train Jeparture, cambridge). destiidttstiiiiinn.mmm cly)
what day would be traveling ? there are 2.828 trains that route. there train that departs friday 9:50 and will arrive in cly at 10:07.
System:
Turn 5
Human:
oops i guess forgot to meation it thursday that noed to travel.
(uttraction-ares area, east), (attraction name, abbey pool and astroturf pitch) (attraction-type swimming pool). (train arriveby. 11:30), (train day, thursday). (train deainn-eprrture. cambridge), (train destination, ely)
Gold Dialog State:
(utrraction-aren east), (attraction-name, abley pool and astrourrf pitch), (train-arrivehy, 11:30),
Predicted Dialog State:
(triin-day thursday). (train departueee cambridge). (triin-dstiiniion, ely)
System:
there are 3 trains that would fit, leaving at 05:50, 07:50, or 09:50
Human:
can get info the 09:50 price the trains id please
Turn 6
(utrraction-ae area, east), (attraction name, abbey pool and astrouuri pitch), (attraction swimming pool).
Gold Dialog State:
(train-snt araiin.rriveyy, 11:30), (train-day, thursday), (train-dpparuurc, cambridge), (traio-destinatinn, cly).
(train neainnle 09:50)
(attraction-area. east), cutraccioo-nanee abbey pool and astroturf pitch), (train-arivvhyy 11:30), (train Jay, thursday). (train cesin-departure.. cambridge), (train- desin-ddsiinionn, (train- 09:50)
Predicted Dialig Stute:
System:
certainly the tmain s id is tr1923, and the price for ricket 4.40 pounds.
Human:
great, thank you that will he all need for (attraction-arca, cast), (attractioo-namc, abbey pool and astroturf pitch). (attraction-type swimming pool). (train trainnarrivcby, 11:30), (train-day, thursday), (train-departure, cambridge), (train-destination, cly),
Turn 7
Gold Dialog State:
(train leaveat, 09:50)
(uttraction area, east),
name, abbey pool astroturf pitch). (train anriveby. 11:30).
Predicted Dialog State:
(train day, thursday). (train departure, cumbridge), (train destination, ely), (train leaveat, 09:50)
System:
are you certain you do not need further assistance?
Human:
9:50 departure 4.40 pounds. tr1923 i got thank
Turn 8
(attraction-aree, east), (attraction- name, abbey pool and astroturf pitch). (attraction-type. swimming pool). (train arriveyy, 11:30), (train day, thursday), (train draartdeearture, cambridge), (tminn destinationn ely).
Gold Dialog State:
(train- leuvea, 09:50)
(attraction-area, cast), nntroottoonnmme abbey pool and sstroturf pitel). (train-arzivcby, 11:30). (train-day, thorsday), (train-departure, cambridge), (trinn-estination, cly', Urrin-keaveat, 09:50)
Predicted Dialog State:
Table 10: Full set of predicted dialogue states for dialogue ID MUL0536 in Mul:WOO2.1.
18

---

Published as a conference paper at ICLR 2020
attraction-area
stractioonname
atraction-name
attraction name
naraction-name
attraction name train-arriveby
train-day rain-departure
destination
train-leaveat
sarract panraana
canaa
attraction monumnne
D'ain-arri elriivee train pran-or day train ren--dpaauure
train train destinationn nationto train-lle -ea---eeeee -eaeeee
attractinn aontoon--e attraction moo--naam
name
attraction-n namee attraction namee
attraction mame
attraction carea
train arrvvvey
day train train-day
train train-depaa deeddenaruee trainddistinaaion restination train leaveat
attraction-el dres attraction tamee mamee
attraction aantton-nameee name
cataroooammtt name
train main-arriveby
train-day
train departure
rainddestraaion restinaation train leaveal
Intractionn aitoo-aaae seraattion--oo coon-nanm
area
namee
attraction name
name
namee
traim arrvvveve
train day
train--de train-depa depeetttttie
attraction
attraction-area
atractioonnamee
attaactionname
atraction-namme
atraction-name
sttraction
trainday
vriin-eparruuee
traindestination
trainleaveat
rame
serracti namee gitzze
sizzztion raine
attrect antractioniii train-arri reveby train day
train deatin-deaatttu dep--eetatte
train wrin-destinn trin-destiratio rainiesveal
attraction area attr perraction name attr attraction nee--temmee attr attraction ierrrcctionn aetttionnaanee attr attractioon-teeeee attra perrcctonnnn name train arively train dryy train train-departun caln-destinaton train leaveat
name
Inrrction-ares attr tttol--ammee namee attr -ettion-naeeee name attr intractioo--ie name attra inrracton-n
area
name
attracto attractionnnel -ame
train-rrmie arrveby
train-dly
train moi--dearture
cain-desiration leavert
attraction area attr Inrractonnn coraano--aaan attr attraction name attracti aoooo----oo
name
train arrvedyy train day arpprrune
sain-destiaation
train-deaart
attr
attraction-area
straction-name
attraction name
straction-name
atraction-name atraction-name trai-aariveby train day
train-degarture train destination train-leaveal
peppottinn ussm
attraciin eettonuaane
baln--amt griveed
train say train departure train destinattonn train--iu sain-leaveed leaveat
attraction areaares straztionn samaaa
atraction uamauaa
attratiinn sorrattonaaat
attraction nane
train train-ariix aeraarrvee arrvebyy train pain-day day
train cete-departuue auturtr
trainddestina ieddesiinaioon nationiio
train leaveat
attraction seraccioomeee erres attractionnaali name attractio-naa mamee
attract intractio-naal name attraction
namm name
altraction eentenn--aaaeee namee
train arrvaeeyeyyy tra m-day train departure
trainddssii inddesiinaionn aatiinni leaveal
attra attraction area attr strration-an tonnnama name attrattion ton camin attrationo namen attraction- alt aatenci iinnnnamma name train serivbeyy train day reparture cin-ddesinattoo
name
attract
attractionaaeea
atraction-namm
atraction-name
attractionname
atraction-name
train-rriveby
vain-degarture
train-day
destination
train-eaveat
att aiaa
aa2z2ti nasin
artzatio nasin
train-s boins eay
train den--eeartutee
train destinatio
train-eeeee eavest
attract attaction toon -tnn---ea attract atraction name aitaatiion 1o namee
area
attraction aottttioonn name name aiitactiootti mammee attraciion rism
train arriveyy
day
aan-eeeetuue denndeettetiinn train leaveat
attraction wewecciioneeeee area att attraction craactoo-naane nome-ammeee name altraction mame
con--mamne attractinn 10 namee
attrackion montonomme name attockion--aul name
train aarreebyyy
traindayy
train departuraet
train destination train leaveat
att iorracctionna alea attr tia aann-namm aftractin in nnmme attrantion cerratttiinnnnn mommaamt raame att aettioniinnnn coo--naan name all attraction firm tame
area name
arryebyy day
train--le dieperture disstination
Figure 5: Visualization of all attention heads in the last attention step T Tstate in the state decoder. The DST prediction is done for the 6th turn dialogue ID MUL0536 MultiWOZ2.1.
19

---

Published as a conference paper at ICLR 2020
Human:
what s your favorite college to visit in cambridge? ?
Turn 1
Gold Dialog State:
(attraction-typc colloge)
Predicted Dialog State:
(ottraction- type, colloge)
like christ callege in the center of town, but there many rthers throughout the city. what part the city did you want
System:
Turn 2
Humun:
think would visit that location, seems nice. could i get the phone number the (attraction-name. christ college), (strraction-typc, college)
Gold Dialog State:
Predieted Dialog State:
(attraction-arca, centre), (attraction-name christ college), (sttraction-typ. college)
System:
the area is centre and the phone number is 01223134900, is there anything else i can help you with actually. i am hungry. can you find me restaurant that serves north american food ? something not super expensive maybe in a moderate price range" ?
Turn 3
Human:
(attraction nume, christ college), (attraction- type, colleges, (restaurant- food, modern american).
Gold Dialog State:
(restaurant-pries prippeocanggg,, moderate)
(attraction-name, christ colloge). (sttraction-type college). trestauanttfood, north american). (restaurant- pricerange, moderate)
Predieted Dialog State:
am sorry. . there restaurant serving specifically north american american food my database, there another type of food you would consider
System:
Turn 4
Human:
how about modern eurppean food
(attraction-name,, christ college), (sttraction-type, collegge), frestaurant- food, modern curopean). (restuuramt- pricerunge, moderate)
Gold Dialog State:
(attraction-name name, christ callege), (attraction- typee, college), frestaurant food, modern european). (restmurant- pricerange, moderate)
Predicted Dialog State:
System:
there are modern europeam restaurant 2 the center and I in the south do you have preference?
Human:
would prefer the on the centre; could have the phone number and postende please
Turn
(attraction-name, christ college)). (attraction-type, college). frestaurantaare, centre).
Gold Dialog State:
(restaurant food, modern europeun), (restaurant (restarrat-priceraagg. moderate)
(attraction nume, christ college). (attraction typpiionn--yyaaaoooe college), (restaurant centre).
Predicted Dialog State:
(restaurant- medemn curopean), (estaurant-riccraang, moderate)
System:
luca cucina and s phone numher 212515566 postende ch2law.
Human:
could you help me get a get from the college the restaurant
Turn 6
(attraction name, christ college), (sttraction-yypr type, college), destaurant-area, centre). (restmuramt-foww food, modern european), (restaurant (restaurat--riccennng pricerange, moderate), (taxi dexarree,, christ college), (taxi destimation, de luca cucina and bar)
Gold Dialog State:
(attraction christ college). (attraction-type, college). (scstauranteerca, contre).
Predicted Dialog State:
(restaurant-foow medern curopcan), (restaurant- pricerange, moderate). (taxi-departure christ college). (taxi- destinnttinn,,,, de loca cucina and bar)
what time would you like to leave the college ? can book you taxi take you the restaurant would like.
System:
Turn 7
Homan:
would like leave by 13:00.
(attraction-nanxx christ college). (sttraction-type. college). tessaauanntareea centre).
Gold Dialog State:
(restaurant- food, modern european). (restaurant pricerange, moderate), (taxi- depanueeeeere,, christ college).
raxiidestinationn. de cucina bar), (taxi-lenveat, 12:45)
(attraction name,, christ collegee), (atraction-type. college). (restaurant-are, centre),
Predicted Dialog State:
(restmurant food, modern european), (restaurant (resauunat-prcerrang.. moderate), (taxi deparure, christ college),
(taxi destination, de luca cucina and bar), (Laxi-leuves, 13:00)
i have booked you taxi leaving at 12:45, the car will be a red toyota and contact uumbar is 07350033543
System:
anything else today? ?
Turn 8
Human:
that s it. thank you very much.
nntrrationnnamsc christ college), (ottraction- type, college). (restaurant centre).
Gold Dialog State:
(restaurant food, medern european), (restaurant prestannatteiiccraage,, moderate). (taxi-deparure, christ college),
(taxi-destinns doidddetination, de loca cucina and bar), (taxi-leavest, 12:45)
(attraction-name, christ college). (sttraction-type. college), trestaurnntarren, centre).
Predicted Dialog State:
(restaurant-fook modern curopcan). cestarrant-ricccaang. moderate). (taxi-dopartre,, christ college).
(tax destination, de cucina bar), (taxi-lenveat, 12:45)
System:
will you need snymore information conserning your stay? ?
Human:
that is all, thanks for the help.
Turn 9
(attraction-nans christ college).. (attraction-tye) college). trestaurantaare, centre),
Gold Dialog State:
(restaurant-food, modern curopean), (restaurant- pesuuraatt-piccranng,, moderate), (taxi-departure, christ christ college),
(taxi destination, de luca cucina and bar), (taxi leuveat, 12:45)
(attraction nume, christ college), (attrattoon type, college). (restaurant-aren, centre).
Predicted Dialog State:
(restaurant-fou medern european), restaurant-ricccragg moderato). (taxi-doparure,, christ college), (taxi- destination, de luca cucina and bar), (taxi- taxi-leavest, 12:45)
Table 11: Full set of predicted dialogue states for dialogue ID PMUL3759 in MultWOZ2.1.
20

---

Published as a conference paper at ICLR 2020
con to
P Wile aatt aocee the pereonn raplava ta Comaaoii
Sare ao
SECOUE and
with Ioe morttn
dost ait
deeeeii eltn
tec seriin las
ao IntEATTTOn Ceeetti ted the Aent aosr
tas dest cnsttatio
and seccraaii
to
di poth ant
THOL
Figure 6: Visualization of all attention heads in the last attention step T,tate the state decoder. The DST prediction is done for the 6th turn in dialogue ID PMUL3759 in MultiWOZ2.1.
21


Reviews:
[{"review_id": "SJltpFKaFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Contribution summary]\nAuthors propose a new model for the DST task that (1) reduces the inference time complexity with an non-autoregressive decoder, and (2) obtains the competitive DST accuracy (49.04% joint accuracy on MultiWoZ 2.1).\n\n[Comments]\n- The proposed model is well motivated and well structured. Empirical results show improvement over other baselines, with the main gain coming from delexicalization, slot gating, fertility output, etc.\n\n- Some of the details are not entirely provided - e.g. please provide the loss hyper-parameter values (e.g. Eq.23) and optimizer parameters for the training.\n\n- Overall presentation, notations, figures, etc. could improve. \n\n- There have been recent work on DST with new SOTA results (e.g. “Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset” by Rastogi et al.) -- please consider comparing the approaches.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HygVkzEAFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a model that is capable of tracking dialogue states in a non-recursive fashion. The main techniques behind the non-recursive model is similar to that of the ICLR 2018 paper \"NON-AUTOREGRESSIVE NEURAL MACHINE TRANSLATION\". Unfortunately, as state tacking can be formulated as one special case of sequence decoding, there is not much of innovation that can be claimed in this paper considering the \"fertility\" idea was already been proposed. The paper did illustrate a strong experimental results on a recent dataset comparing with many state-of-the-art models. However, it is not clear how much innovation this work generates and how the ICLR community would benefit from the problem that the paper is addressing.  \n\n ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJe0EhvPcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors build on recent work for non-autoregressive encoder-decoder models in the context of machine translation (most significantly [Gu, et al., ICLR18]) and adapt this to dialogue state tracking. Specifically, as in [Gu, et al, ICLR18], they use a fertility decoder modified for DST to be on a per-slot basis which is input to a second decoder to generate the (open-vocabulary) tokens representing dialogue state. An interesting aspect of the resulting model as formulated is that the latent space also takes into account interdependencies between generated slot values, which leads to a direct structured prediction-like loss of joint accuracy. Additionally, as slot values have a smaller (and likely peakier) combinatorial space, NAT models actually are more applicable to DST than MT. The resulting model achieves state-of-the-art empirical results on the MultiWOZ dataset while incurring decoding times that are an order of magnitude faster. \n\nOnce one understands this conceptual modification of modifying the NAT string encoder-decoder to a more structured NAT encoder-decoder (which in DST is more of a normalized string), they apply all of the state-of-the-art techniques to build a DST system: gating of all potential (domain, slot) pairs as an auxiliary prediction (e.g., [Wu et al., ACL19]), de-lexicalizing defined value types [Lei, et al., ACL18], content encoder, and domain-slot encoder (with pretty standard hyper-parameters, etc.). The significant addition is the fertility decoder and the associated NAT state decoder. Thus, from a conceptual level, this isn’t a huge leap and something many researchers *could* have done (i.e., I think many people, including myself, to have expected this paper to come out) — thus, it is more of a ‘done first’ paper than a ‘done unexpectedly’ paper. However, it is done well and the results are convincing and interesting. Given the impressive performance, I expect others to continue building on this work and potentially even influencing people to combine encoder-decoder models with a more structured prediction approach to DST.  Thus, I would prefer to see it accepted if possible.\n\nThat being said, I do have a few questions regarding this work — but these are more questions that might be able to be addressed than actual criticisms per se. First, in Table 5, why without the delexicalized dialogue history does the performance drop from 49.04% to 39.45%? This does not make sense to me as the model is much more complex than TRADE; however, TRADE does not do delexicalization yet achieves 45.6% joint accuracy. Meanwhile, with such complex model, I would expect the model can learn from raw data without delexicalization. Moreover, the proposed method use both previous predictions and *previous system actions* to do delexicalization. Also, the NATMT models don’t do delexicalization (although they have significantly more data). I think the authors should do an ablation study that do not use previous system actions, because this is extra information compared with TRADE — even if delexicalizing. Secondly, another worthy baseline would be an autoregressive decoder (with other blocks of the model remain the same). I’d assume that the performance is slight higher. It is interesting to see the gap, because this gap is the cost to speed up decoder using fertility — even if it is a bit counter-intuitive. If there is no improvement in this setting, then structured prediction in general may make more sense. Honestly, I think more would be interested in the second point than the first.\n\nIn any case, nice paper — well-written, well-motivated, interesting empirical results. The only reason I am recommending ‘weak accept’ is that the novelty is fairly straightforward and the strength is in the execution.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: (Please note that I am basing the meta-review on two reviews plus my own thorough read of the paper)
This paper proposes an interesting adaptation of the non-autoregressive neural encoder-decoder models previously proposed for machine translation to dialog state tracking. Experimental results demonstrate state-of-the-art for the MultiWOZ, multi-domain dialog corpus. The reviewers suggest that while the NA approach is not novel, author's adaptation of the approach to dialog state tracking and detailed experimental analysis are interesting and convincing. Hence I suggest accepting the paper as a poster presentation.
Decision: Accept (Poster)