Domain-slot Relationship Modeling using  a Pre-trained Language Encoder for Multi-Domain Dialogue State Tracking

1 INTRODUCTION . A task-oriented dialogue system is designed to help humans solve tasks by understanding their needs and providing relevant information accordingly . For example , such a system may assist its user with making a reservation at an appropriate restaurant by understanding the user ’ s needs for having a nice dinner . It can also recommend an attraction site to a travelling user , accommodating the user ’ s specific preferences . Dialogue State Tracking ( DST ) is a core component of these taskoriented dialogue systems , which aims to identify the state of the dialogue between the user and the system . DST represents the dialogue state with triplets of the following items : a domain , a slot , a value . A set of { restaurant , price range , cheap } , or of { train , arrive-by , 7:00 pm } are examples of such triplets . Fig . 1 illustrates an example case of the dialogue state during the course of the conversation between the user and the system . Since a dialogue continues for multiple turns of utterances , the DST model should successfully predict the dialogue state at each turn as the conversation proceeds . For multi-domain conversations , the DST model should be able to track dialogue states across different domains and slots . Past research on multi-domain conversations used a placeholder in the model to represent domainslot pairs . A domain-slot pair is inserted into the placeholder in each run , and the model runs repeatedly until it covers all types of the domain-slot pairs . ( Wu et al. , 2019 ; Zhang et al. , 2019 ; Lee et al. , 2019 ) . A DST model generally uses an encoder to extract information from the dialogue context that is relevant to the dialogue state . A typical input for a multi-domain DST model comprises a sequence of the user ’ s and the system ’ s utterances up to the turn t , Xt , and the domain-slot information for domain i and slot j , DiSj . In each run , the model feeds the input for a given domain-slot pair through the encoder . fencoder ( Xt , DiSj ) for i = 1 , · · · , n , j = 1 , · · · , m , ( 1 ) where n and m is the number of domains and slots , respectively . However , because each domain-slot pair is modeled independently , the relationship among the domain-slot pairs can not be learned . For example , if the user first asked for a hotel in a certain place and later asked for a restaurant near that hotel , sharing the information between { hotel , area } and { restaurant , area } would help the model recognize that the restaurant should be in the same area as the hotel . Recent approaches address these issues by modeling the dialogue state of every domain-slot pair in a single run , given a dialogue context ( Chen et al. , 2020 ; Le et al. , 2019 ) . This approach can be represented as follows : fencoder ( Xt , D1S1 , · · · , DnSm ) . ( 2 ) Because the encoder receives all of the domain-slot pairs , the model can factor in the relationship among the domain-slot pairs through the encoding process . For the encoder , these studies used models that are trained from scratch , without pre-training . However , since DST involves natural language text for the dialogue context , using a pre-trained language model can help improve the encoding process . Several studies used BERT ( Devlin et al. , 2019 ) , a pre-trained bidirectional language model , for encoding the dialogue context ( Zhang et al. , 2019 ; Lee et al. , 2019 ; Chao & Lane , 2019 ; Gao et al. , 2019 ) , but did not model the dependencies among different domain-slot pairs . Our approach fills the gap between these previous studies . In this work , we propose a model for multi-domain dialogue state tracking that effectively models the relationship among domain-slot pairs using a pre-trained language encoder . We modify the input structure of BERT , specifically the special token part of it , to adjust it for multi-domain DST . The [ CLS ] token of BERT ( Devlin et al. , 2019 ) is expected to encode the aggregate sequence representation as it runs through BERT , which is used for various downstream tasks such as sentence classification or question answering . This [ CLS ] token can also be used as an aggregate representation for a given dialogue context . However , in a multi-domain dialogue , a single [ CLS ] token has to store information for different domain-slot pairs at the same time . In this respect , we propose to use multiple special tokens , one for each domain-slot pair . Using a separate special token for each domain-slot pair is more effective in storing information for different domains and slots since each token can concentrate on its corresponding domain and slot . We consider two different ways to represent such tokens : DS-merge and DS-split . DS-merge employs a single token to represent a single domain-slot pair . For example , to represent a domain-slot pair of { restaurant , area } , we use a special token DS ( restaurant , area ) . DS-split , on the other hand , employs tokens separately for the domain and slot and then merges them into one to represent a domain-slot pair . For { restaurant , area } , the domain token Drestaurant and the slot token Sarea . is computed separately and then merged . We use { DS } merge and { DS } split to represent the special tokens for DS-merge or DS-split , respectively . Unless it is absolutely necessary to specify whether the tokens are from DS-merge or DS-split , we ’ ll refer to the DS-produced tokens as { DS } tokens , without special distinction , in our descriptions forward . The { DS } tokens , after being encoded by the pre-trained language encoder along with the dialogue context , is used to predict its corresponding domain-slot value for a given dialogue context . 2 RELATED WORKS . Recent work on dialogue state tracking can be largely divided into two groups according to how the slot-values are predicted : fixed-vocabulary and open-vocabulary . The fixed-vocabulary approach , also known as the picklisted-based approach , uses a classification module to predict the dialogue state for each slot from a pre-defined set of candidate values ( Zhong et al. , 2018 ; Nouri & HosseiniAsl , 2018 ; Ramadan et al. , 2018 ; Eric et al. , 2019 ; Lee et al. , 2019 ; Chen et al. , 2020 ) . The openvocabulary approach generates the dialogue state for each domain-slot pair either by using a generative decoder to generate text ( Wu et al. , 2019 ; Hosseini-Asl et al. , 2020 ) or by extracting text spans from the dialogue history ( Gao et al. , 2019 ; Goel et al. , 2019 ; Heck et al. , 2020 ) . There is also an approach to use both picklist-based and span-based methods according to the slot type ( Zhang et al. , 2019 ) . For models that deal with multi-domain dialogue , how they deal with different domain-slot pairs is another way to divide them . The first approach encodes the dialogue context independent of the domain-slot pairs and uses separate modules for each domain-slot pair ( Eric et al. , 2019 ; Gao et al. , 2019 ; Goel et al. , 2019 ; Heck et al. , 2020 ) . The second approach encodes the dialogue context using the domain-slot pair information as the prefix and run the encoder multiple times ( Nouri & Hosseini-Asl , 2018 ; Wu et al. , 2019 ) . Other approaches encode the dialogue context independently but merges it with domain-slot pair information later with a separate fusion module ( Zhong et al. , 2018 ; Ramadan et al. , 2018 ; Lee et al. , 2019 ) . However , none of these models are able to model the relationship among different domain-slot pairs because there is no module that enables the interaction between them . ( Le et al. , 2019 ) and ( Chen et al. , 2020 ) directly models the relationship among different domainslot pairs . ( Le et al. , 2019 ) uses a Fertility decoder to learn potential dependencies across domainslot pairs , but without using a pre-trained language model . Also , their model requires additional data such as system action and delexicalized system responses for its performance . ( Chen et al. , 2020 ) also explicitly models the relationship among different domain-slot pairs by using a Graph Attention Network ( GAT ) ( Veličković et al. , 2018 ) . Schema graphs , which is the relation graph between domains and slots , are utilized for connecting edges in the GAT . Our work is different from these works in that we leverage the power of a pre-trained language encoder for directly modeling the dependencies among different domain-slot pairs . ( Hosseini-Asl et al. , 2020 ) takes a different approach from the others by using multi-task learning that encompasses DST as well as action and response generation with a generative language model GPT-2 ( Radford et al. , 2019 ) . However , since our work is focused on DST , we consider the model that is trained on DST only . In the decoding process , dialogue states for different domain-slot pairs are sequentially generated . 3 PROPOSED METHOD . Our model is composed of three parts . The first is the domain-slot-context ( DSC ) encoder , which encodes the dialogue context along with the special tokens representing domain-slot pairs . Next is slot-gate classifier , which is a preliminary classifier that predicts whether each domain-slot pair is relevant to the dialogue context . The adopted the concept of the slot-gate classifier from ( Wu et al. , 2019 ) and made adjustments to apply to our model . The last is the slot value classifier for predicting the value for each domain-slot pair among the candidate values . In the following descriptions , we assume a dialogue context with a total of T turns . The task is to predict the dialogue state , which are { domain , slot , value } triplets for all domain-slot pairs , for every turn t = 1 , · · · , T , using the dialogue context until each turn . Section 3 show the overview of our proposed model . 3.1 DOMAIN-SLOT-CONTEXT ENCODER . The main structure of our model is the DSC encoder , which uses a pre-trained language to encode the dialogue context along with { DS } tokens . For the pre-trained language encoder , we used ALBERT ( Lan et al. , 2019 ) due to its strong performance on numerous natural language understanding tasks while having fewer parameters compared to other BERT-style encoders . { DS } tokens work like the [ CLS ] token for BERT , encoding information corresponding to its domain-slot pair ( DS-merge ) or domain and slot ( DS-split ) . The set of special tokens for each layout are shown in Eq . ( 3 ) and Eq . ( 4 ) , respectively . In DS-merge , we used special tokens for each individual domain-slot pair . If there are many domain-slot pairs , using this layout can increase the number of special tokens as each domain-slot pair requires a separate special token . In DS-split , we used separate tokens for the domain and slot . To represent a domain-slot pair , we merged the corresponding tokens from each domain and slot by concatenating them . This promotes modeling compositionality , since the same slot token can be used for different domains . These { DS } tokens and the dialogue context are processed through the DSC encoder , which results in each token in { DS } being encoded with contextualized representations according to its domain and slot . { DS } merge = { DS ( domain ( 1 ) , slot ( 1 ) ) , · · · , DS ( domain ( n ) , slot ( m ) ) } ( 3 ) { DS } split = { Ddomain ( 1 ) , · · · , Ddomain ( n ) , Sslot ( 1 ) , · · · , Sslot ( m ) } ( 4 ) Fig . 3 shows the input representation of the DSC encoder . The sequence begins with { DS } tokens . The special token [ CLS ] follows , which encodes the overall information of the dialogue context . For the dialogue context , we added a special token [ SEPu ] to separate each user or system utterance , which is added at the end of each utterance from the user or system . The input ends with a special token [ SEP ] as the end-of-sequence token . 4 types of embeddings are summed up to represent each token embedding . We used the pre-trained word embedding of ALBERT , except for the { DS } tokens , which are randomly initialized . We introduced the token type embedding to differentiate the { DS } tokens , user utterances tokens , and system utterances tokens . For DS-merge , we used a single token type embedding to represent a domain-slot pair , whereas for DS-split , we used two token type embeddings , one for the domain and the other for the slot . We did not apply this embedding for the [ CLS ] token . Position embeddings are also employed from ALBERT , but the index of the positional embedding starts from the [ CLS ] token . We did not use the positional embedding for the { DS } tokens as the order within those tokens is meaningless . Lastly , the segment embedding from ALBERT was used to represent the whole sequence as a single segment , which is the default segment embedding of ALBERT . DSC encoder encodes contextualized embeddings for every input token . However , for the slotgate classifier and slot-value classifier , we only use the special token outputs of the DSC encoder ( [ CLS ] token and { DS } tokens ) . This is formally defined as follows for DS-merge and DS-split , respectively , for turn t : D̂S ( 1,1 ) , · · · , D̂S ( n , m ) , ĈLS = DSCencoder ( [ { DS } merge , CLS , Xt , SEP ] ) , ( 5 ) D̂1 , · · · , D̂n , Ŝ1 · · · , Ŝm , ĈLS = DSCencoder ( [ { DS } split , CLS , Xt , SEP ] ) , ( 6 ) where Xt represents the dialogue context of ( S1 , SEPuU1 , SEPu , · · · , St , SEPu , U t , SEPu ) . U t and St represents the utterance for the tth turn for the user and system respectively . The { DS } tokens and [ CLS ] token with the hat notation ̂ represents the encoded output of the DSC encoder for those special tokens . They are vectors of Rd , where d is the hidden dimension of ALBERT .

Summary: In this paper, the authors proposed a multidomain state-tracking model that leverages the relationship among different domain-slot pairs. This is done by leveraging the full-attention step over the [CLS] special token and by providing all the domain-slot pairs as a special token to a pre-trained language model (Figure 2 is very clear). To predict the value of the slot $D_{i,j}$, the author concatenates the representation of the [CLS] token, share among all the domain-slots, and the $D_{i,j}$, provided as input, and use a gating mechanism, by only using $D_{i,j}$ representation, to decide whether require as value (i.e., prediction) or not (e.g. None). \