Non-Autoregressive Dialog State Tracking

1 INTRODUCTION . In task-oriented dialogues , a dialogue agent is required to assist humans for one or many tasks such as finding a restaurant and booking a hotel . As a sample dialogue shown in Table 1 , each user utterance typically contains important information identified as slots related to a dialogue domain such as attraction-area and train-day . A crucial part of a task-oriented dialogue system is Dialogue State Tracking ( DST ) , which aims to identify user goals expressed during a conversation in the form of dialogue states . A dialogue state consists of a set of ( slot , value ) pairs e.g . ( attraction-area , centre ) and ( train-day , tuesday ) . Existing DST models can be categorized into two types : fixed- and open-vocabulary . Fixed vocabulary models assume known slot ontology and generate a score for each candidate of ( slot , value ) ( Ramadan et al. , 2018 ; Lee et al. , 2019 ) . Recent approaches propose open-vocabulary models that can generate the candidates , especially for slots such as entity names and time , from the dialogue history ( Lei et al. , 2018 ; Wu et al. , 2019 ) . Most open-vocabulary DST models rely on autoregressive encoders and decoders , which encode dialogue history sequentially and generate token ti of individual slot value one by one conditioned on all previously generated tokens t [ 1 : i−1 ] . For downstream tasks of DST that emphasize on low latency ( e.g . generating real-time dialogue responses ) , auto-regressive approaches incur expensive time cost as the ongoing dialogues become more complex . The time cost is caused by two major components : length of dialogue history i.e . number of turns , and length of slot values . For complex dialogues extended over many turns and multiple domains , the time cost will increase significantly in both encoding and decoding phases . Similar problems can be seen in the field of Neural Machine Translation ( NMT ) research where a long piece of text is translated from one language to another . Recent work has tried to improve the ∗All work was done while the first author was a research intern at Salesforce Research Asia . latency in NMT by using neural network architectures such as convolution ( Krizhevsky et al. , 2012 ) and attention ( Luong et al. , 2015 ) . Several non- and semi-autoregressive approaches aim to generate tokens of the target language independently ( Gu et al. , 2018 ; Lee et al. , 2018 ; Kaiser et al. , 2018 ) . Motivated by this line of research , we thus propose a non-autoregressive approach to minimize the time cost of DST models without a negative impact on the model performance . We adopt the concept of fertility proposed by Gu et al . ( 2018 ) . Fertility denotes the number of times each input token is copied to form a sequence as the input to the decoder for non-autoregressive decoding . We first reconstruct dialogue state as a sequence of concatenated slot values . The result sequence contains the inherent structured representation in which we can apply the fertility concept . The structure is defined by the boundaries of individual slot values . These boundaries can be easily obtained from dialogue state itself by simply measuring number of the tokens of individual slots . Our model includes a two-stage decoding process : ( 1 ) the first decoder learns relevant signals from the input dialogue history and generates a fertility for each input slot representation ; and ( 2 ) the predicted fertility is used to form a structured sequence which consists of multiple sub-sequences , each represented as ( slot token×slot fertility ) . The result sequence is used as input to the second decoder to generate all the tokens of the target dialogue state at once . In addition to being non-autoregressive , our models explicitly consider dependencies at both slot level and token level . Most of existing DST models assume independence among slots in dialogue states without explicitly considering potential signals across the slots ( Wu et al. , 2019 ; Lee et al. , 2019 ; Goel et al. , 2019 ; Gao et al. , 2019 ) . However , we hypothesize that it is not true in many cases . For example , a good DST model should detect the relation that train departure should not have the same value as train destination ( example in Table 1 ) . Other cases include time-related pairs such as ( taxi arriveBy , taxi leaveAt ) and cross-domain pairs such as ( hotel area , attraction area ) . Our proposed approach considers all possible signals across all domains and slots to generate a dialogue state as a set . Our approach directly optimizes towards the DST evaluation metric Joint Accuracy ( Henderson et al. , 2014b ) , which measures accuracy at state ( set of slots ) level rather than slot level . Our contributions in this work include : ( 1 ) we propose a novel framework of Non-Autoregressive Dialog State Tracking ( NADST ) , which explicitly learns inter-dependencies across slots for decoding dialogue states as a complete set rather than individual slots ; ( 2 ) we propose a non-autoregressive decoding scheme , which not only enjoys low latency for real-time dialogues , but also allows to capture dependencies at token level in addition to slot level ; ( 3 ) we achieve the state-of-the-art performance on the multi-domain task-oriented dialogue dataset “ MultiWOZ 2.1 ” ( Budzianowski et al. , 2018 ; Eric et al. , 2019 ) while significantly reducing the inference latency by an order of magnitude ; ( 4 ) we conduct extensive ablation studies in which our analysis reveals that our models can detect potential signals across slots and dialogue domains to generate more correct “ sets ” of slots for DST . 2 RELATED WORK . Our work is related to two research areas : dialogue state tracking and non-autoregressive decoding . 2.1 DIALOGUE STATE TRACKING . Dialogue State Tracking ( DST ) is an important component in task-oriented dialogues , especially for dialogues with complex domains that require fine-grained tracking of relevant slots . Traditionally , DST is coupled with Natural Language Understanding ( NLU ) . NLU output as tagged user utterances is input to DST models to update the dialogue states turn by turn ( Kurata et al. , 2016 ; Shi et al. , 2016 ; Rastogi et al. , 2017 ) . Recent approaches combine NLU and DST to reduce the credit assignment problem and remove the need for NLU ( Mrkšić et al. , 2017 ; Xu & Hu , 2018 ; Zhong et al. , 2018 ) . Within this body of research , Goel et al . ( 2019 ) differentiates two DST approaches : fixed- and openvocabulary . Fixed-vocabulary approaches are usually retrieval-based methods in which all candidate pairs of ( slot , value ) from a given slot ontology are considered and the models predict a probability score for each pair ( Henderson et al. , 2014c ; Ramadan et al. , 2018 ; Lee et al. , 2019 ) . Recent work has moved towards open-vocabulary approaches that can generate the candidates based on input text i.e . dialogue history ( Lei et al. , 2018 ; Gao et al. , 2019 ; Wu et al. , 2019 ) . Our work is more related to these models , but different from most of the current work , we explicitly consider dependencies among slots and domains to decode dialogue state as a complete set . 2.2 NON-AUTOREGRESSIVE DECODING . Most of prior work in non- or semi-autoregressive decoding methods are used for NMT to address the need for fast translation . Schwenk ( 2012 ) proposes to estimate the translation model probabilities of a phase-based NMT system . Libovickỳ & Helcl ( 2018 ) formulates the decoding process as a sequence labeling task by projecting source sequence into a longer sequence and applying CTC loss ( Graves et al. , 2006 ) to decode the target sequence . Wang et al . ( 2019 ) adds regularization terms to NAT models ( Gu et al. , 2018 ) to reduce translation errors such as repeated tokens and incomplete sentences . Ghazvininejad et al . ( 2019 ) uses a non-autoregressive decoder with masked attention to decode target sequences over multiple generation rounds . A common challenge in nonautoregressive NMT is the large number of sequential latent variables , e.g. , fertility sequences ( Gu et al. , 2018 ) and projected target sequences ( Libovickỳ & Helcl , 2018 ) . These latent variables are used as supporting signals for non- or semi-autoregressive decoding . We reformulate dialogue state as a structured sequence with sub-sequences defined as a concatenation of slot values . This form of dialogue state can be inferred easily from the dialogue state annotation itself whereas such supervision information is not directly available in NMT . The lower semantic complexity of slot values as compared to long sentences in NMT makes it easier to adopt non-autoregressive approaches into DST . According to our review , we are the first to apply a non-autoregressive framework for generation-based DST . Our approach allows joint state tracking across slots , which results in better performance and an order of magnitude lower latency during inference . 3 APPROACH . Our NADST model is composed of three parts : encoders , fertility decoder , and state decoder , as shown in Figure 1 . The input includes the dialogue history X = ( x1 , ... , xN ) and a sequence of applicable ( domain , slot ) pairsXds = ( ( d1 , s1 ) , ... , ( dG , sH ) ) , whereG andH are the total numbers of domains and slots , respectively . The output is the corresponding dialogue states up to the current dialogue history . Conventionally , the output of dialogue state is denoted as tuple ( slot , value ) ( or ( domain-slot , value ) for multi-domain dialogues ) . We reformulate the output as a concatenation of slot values Y di , sj : Y = ( Y d1 , s1 , ... , Y dI , sJ ) = ( yd1 , s11 , y d1 , s1 2 , ... , y dI , sJ 1 , y dI , sJ 2 , ... ) where I and J are the numbers of domains and slots in the output dialogue state , respectively . First , the encoders use token-level embedding and positional encoding to encode the input dialogue history and ( domain , slot ) pairs into continuous representations . The encoded domains and slots are then input to stacked self-attention and feed-forward network to obtain relevant signals across dialogue history and generate a fertility Y dg , shf for each ( domain , slot ) pair ( dg , sh ) . The output of fertility decoder is defined as a sequence : Yfert = Y d1 , s1 f , ... , Y dG , sH f where Y dg , dh f ∈ { 0 , max ( SlotLength ) } . For example , for the MultiWOZ dataset in our experiments , we have max ( SlotLength ) = 9 according to the training data . We follow ( Wu et al. , 2019 ; Gao et al. , 2019 ) to add a slot gating mechanism as an auxiliary prediction . Each gate g is restricted to 3 possible values : “ none ” , “ dontcare ” and “ generate ” . They are used to form higher-level classification signals to support fertility decoding process . The gate output is defined as a sequence : Ygate = Y d1 , s1g , ... , Y dG , sH g . The predicted fertilities are used to form an input sequence to the state decoder for nonautoregressive decoding . The sequence includes sub-sequences of ( dg , sh ) repeated by Y dg , sh f times and concatenated sequentially : Xds×fert = ( ( d1 , s1 ) Y d1 , s1 f , ... , ( dG , sH ) Y dG , sH f ) and ‖Xds×fert‖ = ‖Y ‖ . The decoder projects this sequence through attention layers with dialogue history . During this decoding process , we maintain a memory of hidden states of dialogue history . The output from the state decoder is used as a query to attend on this memory and copy tokens from the dialogue history to generate a dialogue state . Following Lei et al . ( 2018 ) , we incorporate information from previous dialogue turns to predict current turn state by using a partially delexicalized dialogue historyXdel = ( x1 , del , ... , xN , del ) as an input of the model . The dialogue history is delexicalized till the last system utterance by removing real-value tokens that match the previously decoded slot values to tokens expressed as domain-slot . Given a token xn and the current dialogue turn t , the token is delexicalized as follows : xn , del =delex ( xn ) = { domainidx-slotidx , if xn ⊂ Ŷt−1 . xn , otherwise . ( 1 ) domainidx = Xds×fert [ idx ] [ 0 ] , slotidx = Xds×fert [ idx ] [ 1 ] , idx = Index ( xn , Ŷt−1 ) ( 2 ) For example , the user utterance “ I look for a cheap hotel ” is delexicalized to “ I look for a hotel pricerange hotel. ” if the slot hotel pricerange is predicted as “ cheap ” in the previous turn . This approach makes use of the delexicalized form of dialogue history while not relying on an NLU module as we utilize the predicted state from DST model itself . In addition to the belief state , we also use the system action in the previous turn to delexicalize the dialog history in a similar manner , following prior work ( Rastogi et al. , 2017 ; Zhong et al. , 2018 ; Goel et al. , 2019 ) .

Summary: This paper proposed a model that is capable of tracking dialogue states in a non-recursive fashion. The main techniques behind the non-recursive model is similar to that of the ICLR 2018 paper "NON-AUTOREGRESSIVE NEURAL MACHINE TRANSLATION". Unfortunately, as state tacking can be formulated as one special case of sequence decoding, there is not much of innovation that can be claimed in this paper considering the "fertility" idea was already been proposed. The paper did illustrate a strong experimental results on a recent dataset comparing with many state-of-the-art models. However, it is not clear how much innovation this work generates and how the ICLR community would benefit from the problem that the paper is addressing.