Graph Convolutional Network with Sequential Attention For Goal-Oriented Dialogue Systems

Domain specific goal-oriented dialogue systems typically require modeling three types of inputs , viz. , ( i ) the knowledge-base associated with the domain , ( ii ) the history of the conversation , which is a sequence of utterances and ( iii ) the current utterance for which the response needs to be generated . While modeling these inputs , current state-of-the-art models such as Mem2Seq typically ignore the rich structure inherent in the knowledge graph and the sentences in the conversation context . Inspired by the recent success of structure-aware Graph Convolutional Networks ( GCNs ) for various NLP tasks such as machine translation , semantic role labeling and document dating , we propose a memory augmented GCN for goal-oriented dialogues . Our model exploits ( i ) the entity relation graph in a knowledge-base and ( ii ) the dependency graph associated with an utterance to compute richer representations for words and entities . Further , we take cognizance of the fact that in certain situations , such as , when the conversation is in a code-mixed language , dependency parsers may not be available . We show that in such situations we could use the global word co-occurrence graph and use it to enrich the representations of utterances . We experiment with the modified DSTC2 dataset and its recently released code-mixed versions in four languages and show that our method outperforms existing state-of-the-art methods , using a wide range of evaluation metrics . 1 INTRODUCTION . Goal-oriented dialogue systems which can assist humans in various day-to-day activities have widespread applications in several domains such as e-commerce , entertainment , healthcare , etc . For example , such systems can help humans in scheduling medical appointments , reserving restaurants , booking tickets , etc .. From a modeling perspective , one clear advantage of dealing with domain specific goal-oriented dialogues is that the vocabulary is typically limited , the utterances largely follow a fixed set of templates and there is an associated domain knowledge which can be exploited . More specifically , there is some structure associated with the utterances as well as the knowledge base . More formally , the task here is to generate the next response given ( i ) the previous utterances in the conversation history ( ii ) the current user utterance ( known as the query ) and ( iii ) the entities and relationships in the associated knowledge base . Current state-of-the-art methods ( Seo et al. , 2017 ; Eric & Manning , 2017 ; Madotto et al. , 2018 ) typically use variants of Recurrent Neural Network ( Elman , 1990 ) to encode the history and current utterance and an external memory network to store the entities in the knowledge base . The encodings of the utterances and memory elements are then suitably combined using an attention network and fed to the decoder to generate the response , one word at a time . However , these methods do not exploit the structure in the knowledge base as defined by entity-entity relations and the structure in the utterances as defined by a dependency parse . Such structural information can be exploited to improve the performance of the system as demonstrated by recent works on syntax-aware neural machine translation ( Eriguchi et al. , 2016 ; Bastings et al. , 2017 ; Chen et al. , 2017 ) , semantic role labeling ( Marcheggiani & Titov , 2017 ) and document dating ( Vashishth et al. , 2018 ) which use GCNs ( Defferrard et al. , 2016 ; Duvenaud et al. , 2015 ; Kipf & Welling , 2017 ) to exploit sentence structure . In this work , we propose to use such graph structures for goal-oriented dialogues . In particular , we compute the dependency parse tree for each utterance in the conversation and use a GCN to capture the interactions between words . This allows us to capture interactions between distant words in the sentence as long as they are connected by a dependency relation . We also use GCNs to encode the entities of the KB where the entities are treated as nodes and the relations as edges of the graph . Once we have a richer structure aware representation for the utterances and the entities , we use a sequential attention mechanism to compute an aggregated context representation from the GCN node vectors of the query , history and entities . Further , we note that in certain situations , such as , when the conversation is in a code-mixed language or a language for which parsers are not available then it may not be possible to construct a dependency parse for the utterances . To overcome this , we construct a co-occurrence matrix from the entire corpus and use this matrix to impose a graph structure on the utterances . More specifically , we add an edge between two words in a sentence if they co-occur frequently in the corpus . Our experiments suggest that this simple strategy acts as a reasonable substitute for dependency parse trees . We perform experiments with the modified DSTC2 ( Bordes et al. , 2017 ) dataset which contains goal-oriented conversations for reserving restaurants . We also use its recently released code-mixed versions ( Banerjee et al. , 2018 ) which contain code-mixed conversations in four different languages , viz. , Hindi , Bengali , Gujarati and Tamil . We compare with recent state-of-the-art methods and show that on average the proposed model gives an improvement of 2.8 BLEU points and 2 ROUGE points . Our contributions can be summarized as follows : ( i ) We use GCNs to incorporate structural information for encoding query , history and KB entities in goal-oriented dialogues ( ii ) We use a sequential attention mechanism to obtain query aware and history aware context representations ( iii ) We leverage co-occurrence frequencies and PPMI ( positive-pointwise mutual information ) values to construct contextual graphs for code-mixed utterances and ( iv ) We show that the proposed model obtains state-of-the-art results on the modified DSTC2 dataset and its recently released code-mixed versions . 2 RELATED WORK . In this section we review the previous work in goal-oriented dialogue systems and describe the introduction of GCNs in NLP . Goal-Oriented Dialogue System : Initial goal-oriented dialogue systems ( Young , 2000 ; Williams & Young , 2007 ) were based on dialogue state tracking ( Williams et al. , 2013 ; Henderson et al. , 2014a ; b ) and included pipelined modules for natural language understanding , dialogue state tracking , policy management and natural language generation . Wen et al . ( 2017 ) used neural networks for these intermediate modules but still lacked absolute end-to-end trainability . Such pipelined modules were restricted by the fixed slot-structure assumptions on the dialogue state and required per-module based labelling . To mitigate this problem Bordes et al . ( 2017 ) released a version of goal-oriented dialogue dataset that focuses on the development of end-to-end neural models . Such models need to reason over the associated KB triples and generate responses directly from the utterances without any additional annotations . For example , Bordes et al . ( 2017 ) proposed a Memory Network ( Sukhbaatar et al. , 2015 ) based model to match the response candidates with the multi-hop attention weighted representation of the conversation history and the KB triples in memory . Liu & Perez ( 2017 ) further added highway ( Srivastava et al. , 2015 ) and residual connections ( He et al. , 2016 ) to the memory network in order to regulate the access to the memory blocks . Seo et al . ( 2017 ) developed a variant of RNN cell which computes a refined representation of the query over multiple iterations before querying the memory . However , all these approaches retrieve the response from a set of candidate responses and such a candidate set is not easy to obtain in any new domain of interest . To account for this , Eric & Manning ( 2017 ) ; Zhao et al . ( 2017 ) adapted RNN based encoder-decoder models to generate appropriate responses instead of retrieving them from a candidate set . Eric et al . ( 2017 ) introduced a key-value memory network based generative model which integrates the underlying KB with RNN based encode-attend-decode models . Madotto et al . ( 2018 ) used memory networks on top of the RNN decoder to tightly integrate KB entities with the decoder to generate more infor- mative responses . However , as opposed to our work , all these works ignore the underlying structure of the entity-relation graph of the KB and the syntactic structure of the utterances . GCNs in NLP : Recently , there has been an active interest in enriching existing encode-attenddecode models ( Bahdanau et al. , 2015 ) with structural information for various NLP tasks . Such structure is typically obtained from the constituency and/or dependency parse of sentences . The idea is to treat the output of a parser as a graph and use an appropriate network to capture the interactions between the nodes of this graph . For example , Eriguchi et al . ( 2016 ) and Chen et al . ( 2017 ) showed that incorporating such syntactical structures as Tree-LSTMs in the encoder can improve the performance of Neural Machine Translation ( NMT ) . Peng et al . ( 2017 ) use Graph-LSTMs to perform cross sentence n-ary relation extraction and show that their formulation is applicable to any graph structure and Tree-LSTMs can be thought of as a special case of it . In parallel , Graph Convolutional Networks ( GCNs ) ( Duvenaud et al. , 2015 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ) and their variants ( Li et al. , 2015 ) have emerged as state-of-the-art methods for computing representations of entities in a knowledge graph . They provide a more flexible way of encoding such graph structures by capturing multi-hop relationships between nodes . This has led to their adoption for various NLP tasks such as neural machine translation ( Marcheggiani et al. , 2018 ; Bastings et al. , 2017 ) , semantic role labeling ( Marcheggiani & Titov , 2017 ) , document dating ( Vashishth et al. , 2018 ) and question answering ( Johnson , 2017 ; Nicola De Cao , 2018 ) . To the best of our knowledge ours is the first work that uses GCNs to incorporate dependency structural information and the entity-entity graph structure in a single end-to-end neural model for goaloriented dialogue . This is also the first work that incorporates contextual co-occurrence information for code-mixed utterances , for which no dependency structures are available . 3 BACKGROUND . In this section we describe Graph Convolutional Networks ( GCN ) ( Kipf & Welling , 2017 ) for undirected graphs and then describe their syntactic versions which work with directed labeled edges of dependency parse trees . 3.1 GCN FOR UNDIRECTED GRAPHS . Graph convolutional networks operate on a graph structure and compute representations for the nodes of the graph by looking at the neighbourhood of the node . k layers of GCNs can be stacked to account for neighbours which are k-hops away from the current node . Formally , let G = ( V , E ) be an undirected graph where V is the set of nodes ( let |V| = n ) and E is the set of edges . Let X ∈ Rn×m be the input feature matrix with n nodes and each node xu ( u ∈ V ) is represented by an m-dimensional feature vector . The output of a 1-layer GCN is the hidden representation matrix H ∈ Rn×d where each d-dimensional representation of a node captures the interactions with its 1-hop neighbour . Each row of this matrix can be computed as : hv = ReLU ( ∑ u∈N ( v ) ( Wxu + b ) ) , ∀v ∈ V ( 1 ) Here W ∈ Rd×m is the model parameter matrix , b ∈ Rd is the bias vector and ReLU is the rectified linear unit activation function . N ( v ) is the set of neighbours of node v and is assumed to also include the node v so that the previous representation of the node v is also considered while computing the new hidden representation . To capture interactions with nodes which are multiple hops away , multiple layers of GCNs can be stacked together . Specifically , the representation of node v after kth GCN layer can be formulated as : hk+1v = ReLU ( ∑ u∈N ( v ) ( W khku + b k ) ) , ∀v ∈ V ( 2 ) where hku is the representation of the u th node in the ( k − 1 ) th GCN layer and h1u = xu .

Summary: The paper proposes a Graph Convolutional Network-based encoder-decoder model with sequential attention for goal-oriented dialogue systems, with the purpose of exploiting the graph structures in KB and sentences in conversation. The model consists of three encoders for a query, dialogue history, and KB, respectively, and a decoder with a sequential attention mechanism. The proposed model attains state-of-the-art performance on the modified DSTC2 dataset of (Bordes et al., 2017). For the experiments with graphs constructed from word co-occurrence matrix, code-mixed versions of modified DSTC2 released by (Banerjee et al., 2018) are used.