Graph Convolutional Network with Sequential Attention For Goal-Oriented Dialogue Systems

Domain specific goal-oriented dialogue systems typically require modeling three types of inputs , viz. , ( i ) the knowledge-base associated with the domain , ( ii ) the history of the conversation , which is a sequence of utterances and ( iii ) the current utterance for which the response needs to be generated . While modeling these inputs , current state-of-the-art models such as Mem2Seq typically ignore the rich structure inherent in the knowledge graph and the sentences in the conversation context . Inspired by the recent success of structure-aware Graph Convolutional Networks ( GCNs ) for various NLP tasks such as machine translation , semantic role labeling and document dating , we propose a memory augmented GCN for goal-oriented dialogues . Our model exploits ( i ) the entity relation graph in a knowledge-base and ( ii ) the dependency graph associated with an utterance to compute richer representations for words and entities . Further , we take cognizance of the fact that in certain situations , such as , when the conversation is in a code-mixed language , dependency parsers may not be available . We show that in such situations we could use the global word co-occurrence graph and use it to enrich the representations of utterances . We experiment with the modified DSTC2 dataset and its recently released code-mixed versions in four languages and show that our method outperforms existing state-of-the-art methods , using a wide range of evaluation metrics . 1 INTRODUCTION . Goal-oriented dialogue systems which can assist humans in various day-to-day activities have widespread applications in several domains such as e-commerce , entertainment , healthcare , etc . For example , such systems can help humans in scheduling medical appointments , reserving restaurants , booking tickets , etc .. From a modeling perspective , one clear advantage of dealing with domain specific goal-oriented dialogues is that the vocabulary is typically limited , the utterances largely follow a fixed set of templates and there is an associated domain knowledge which can be exploited . More specifically , there is some structure associated with the utterances as well as the knowledge base . More formally , the task here is to generate the next response given ( i ) the previous utterances in the conversation history ( ii ) the current user utterance ( known as the query ) and ( iii ) the entities and relationships in the associated knowledge base . Current state-of-the-art methods ( Seo et al. , 2017 ; Eric & Manning , 2017 ; Madotto et al. , 2018 ) typically use variants of Recurrent Neural Network ( Elman , 1990 ) to encode the history and current utterance and an external memory network to store the entities in the knowledge base . The encodings of the utterances and memory elements are then suitably combined using an attention network and fed to the decoder to generate the response , one word at a time . However , these methods do not exploit the structure in the knowledge base as defined by entity-entity relations and the structure in the utterances as defined by a dependency parse . Such structural information can be exploited to improve the performance of the system as demonstrated by recent works on syntax-aware neural machine translation ( Eriguchi et al. , 2016 ; Bastings et al. , 2017 ; Chen et al. , 2017 ) , semantic role labeling ( Marcheggiani & Titov , 2017 ) and document dating ( Vashishth et al. , 2018 ) which use GCNs ( Defferrard et al. , 2016 ; Duvenaud et al. , 2015 ; Kipf & Welling , 2017 ) to exploit sentence structure . In this work , we propose to use such graph structures for goal-oriented dialogues . In particular , we compute the dependency parse tree for each utterance in the conversation and use a GCN to capture the interactions between words . This allows us to capture interactions between distant words in the sentence as long as they are connected by a dependency relation . We also use GCNs to encode the entities of the KB where the entities are treated as nodes and the relations as edges of the graph . Once we have a richer structure aware representation for the utterances and the entities , we use a sequential attention mechanism to compute an aggregated context representation from the GCN node vectors of the query , history and entities . Further , we note that in certain situations , such as , when the conversation is in a code-mixed language or a language for which parsers are not available then it may not be possible to construct a dependency parse for the utterances . To overcome this , we construct a co-occurrence matrix from the entire corpus and use this matrix to impose a graph structure on the utterances . More specifically , we add an edge between two words in a sentence if they co-occur frequently in the corpus . Our experiments suggest that this simple strategy acts as a reasonable substitute for dependency parse trees . We perform experiments with the modified DSTC2 ( Bordes et al. , 2017 ) dataset which contains goal-oriented conversations for reserving restaurants . We also use its recently released code-mixed versions ( Banerjee et al. , 2018 ) which contain code-mixed conversations in four different languages , viz. , Hindi , Bengali , Gujarati and Tamil . We compare with recent state-of-the-art methods and show that on average the proposed model gives an improvement of 2.8 BLEU points and 2 ROUGE points . Our contributions can be summarized as follows : ( i ) We use GCNs to incorporate structural information for encoding query , history and KB entities in goal-oriented dialogues ( ii ) We use a sequential attention mechanism to obtain query aware and history aware context representations ( iii ) We leverage co-occurrence frequencies and PPMI ( positive-pointwise mutual information ) values to construct contextual graphs for code-mixed utterances and ( iv ) We show that the proposed model obtains state-of-the-art results on the modified DSTC2 dataset and its recently released code-mixed versions . 2 RELATED WORK . In this section we review the previous work in goal-oriented dialogue systems and describe the introduction of GCNs in NLP . Goal-Oriented Dialogue System : Initial goal-oriented dialogue systems ( Young , 2000 ; Williams & Young , 2007 ) were based on dialogue state tracking ( Williams et al. , 2013 ; Henderson et al. , 2014a ; b ) and included pipelined modules for natural language understanding , dialogue state tracking , policy management and natural language generation . Wen et al . ( 2017 ) used neural networks for these intermediate modules but still lacked absolute end-to-end trainability . Such pipelined modules were restricted by the fixed slot-structure assumptions on the dialogue state and required per-module based labelling . To mitigate this problem Bordes et al . ( 2017 ) released a version of goal-oriented dialogue dataset that focuses on the development of end-to-end neural models . Such models need to reason over the associated KB triples and generate responses directly from the utterances without any additional annotations . For example , Bordes et al . ( 2017 ) proposed a Memory Network ( Sukhbaatar et al. , 2015 ) based model to match the response candidates with the multi-hop attention weighted representation of the conversation history and the KB triples in memory . Liu & Perez ( 2017 ) further added highway ( Srivastava et al. , 2015 ) and residual connections ( He et al. , 2016 ) to the memory network in order to regulate the access to the memory blocks . Seo et al . ( 2017 ) developed a variant of RNN cell which computes a refined representation of the query over multiple iterations before querying the memory . However , all these approaches retrieve the response from a set of candidate responses and such a candidate set is not easy to obtain in any new domain of interest . To account for this , Eric & Manning ( 2017 ) ; Zhao et al . ( 2017 ) adapted RNN based encoder-decoder models to generate appropriate responses instead of retrieving them from a candidate set . Eric et al . ( 2017 ) introduced a key-value memory network based generative model which integrates the underlying KB with RNN based encode-attend-decode models . Madotto et al . ( 2018 ) used memory networks on top of the RNN decoder to tightly integrate KB entities with the decoder to generate more infor- mative responses . However , as opposed to our work , all these works ignore the underlying structure of the entity-relation graph of the KB and the syntactic structure of the utterances . GCNs in NLP : Recently , there has been an active interest in enriching existing encode-attenddecode models ( Bahdanau et al. , 2015 ) with structural information for various NLP tasks . Such structure is typically obtained from the constituency and/or dependency parse of sentences . The idea is to treat the output of a parser as a graph and use an appropriate network to capture the interactions between the nodes of this graph . For example , Eriguchi et al . ( 2016 ) and Chen et al . ( 2017 ) showed that incorporating such syntactical structures as Tree-LSTMs in the encoder can improve the performance of Neural Machine Translation ( NMT ) . Peng et al . ( 2017 ) use Graph-LSTMs to perform cross sentence n-ary relation extraction and show that their formulation is applicable to any graph structure and Tree-LSTMs can be thought of as a special case of it . In parallel , Graph Convolutional Networks ( GCNs ) ( Duvenaud et al. , 2015 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ) and their variants ( Li et al. , 2015 ) have emerged as state-of-the-art methods for computing representations of entities in a knowledge graph . They provide a more flexible way of encoding such graph structures by capturing multi-hop relationships between nodes . This has led to their adoption for various NLP tasks such as neural machine translation ( Marcheggiani et al. , 2018 ; Bastings et al. , 2017 ) , semantic role labeling ( Marcheggiani & Titov , 2017 ) , document dating ( Vashishth et al. , 2018 ) and question answering ( Johnson , 2017 ; Nicola De Cao , 2018 ) . To the best of our knowledge ours is the first work that uses GCNs to incorporate dependency structural information and the entity-entity graph structure in a single end-to-end neural model for goaloriented dialogue . This is also the first work that incorporates contextual co-occurrence information for code-mixed utterances , for which no dependency structures are available . 3 BACKGROUND . In this section we describe Graph Convolutional Networks ( GCN ) ( Kipf & Welling , 2017 ) for undirected graphs and then describe their syntactic versions which work with directed labeled edges of dependency parse trees . 3.1 GCN FOR UNDIRECTED GRAPHS . Graph convolutional networks operate on a graph structure and compute representations for the nodes of the graph by looking at the neighbourhood of the node . k layers of GCNs can be stacked to account for neighbours which are k-hops away from the current node . Formally , let G = ( V , E ) be an undirected graph where V is the set of nodes ( let |V| = n ) and E is the set of edges . Let X ∈ Rn×m be the input feature matrix with n nodes and each node xu ( u ∈ V ) is represented by an m-dimensional feature vector . The output of a 1-layer GCN is the hidden representation matrix H ∈ Rn×d where each d-dimensional representation of a node captures the interactions with its 1-hop neighbour . Each row of this matrix can be computed as : hv = ReLU ( ∑ u∈N ( v ) ( Wxu + b ) ) , ∀v ∈ V ( 1 ) Here W ∈ Rd×m is the model parameter matrix , b ∈ Rd is the bias vector and ReLU is the rectified linear unit activation function . N ( v ) is the set of neighbours of node v and is assumed to also include the node v so that the previous representation of the node v is also considered while computing the new hidden representation . To capture interactions with nodes which are multiple hops away , multiple layers of GCNs can be stacked together . Specifically , the representation of node v after kth GCN layer can be formulated as : hk+1v = ReLU ( ∑ u∈N ( v ) ( W khku + b k ) ) , ∀v ∈ V ( 2 ) where hku is the representation of the u th node in the ( k − 1 ) th GCN layer and h1u = xu .

Summary: The current paper proposes using Graph Convolutional Networks (GCN) to explicitly represent and use relational data in dialog modeling, as well an attention mechanism for combining information from multiple sources (dialog history, knowledge base, current utterance). The work assumes that the knowledge base associated with the dialog task has en entity-to-entity-relationship format and can be naturally expressed as a graph. The dependency tree of dialog utterances can also be expressed as a graph, and the dialog history as a set of graphs. To utilize this structure, the proposed method uses GCNs whose lowest layer embeddings are initialized with the entity embeddings or via outputs of standard RNN-like models. The main claim is that the proposed model outperforms the current state-of-the-art on a goal-oriented dialog task.