Language Modeling with Graph Temporal Convolutional Networks

1 INTRODUCTION . Recurrent neural networks ( RNNs ) are currently considered the de facto tool for language modeling tasks . RNNs encode sentences by processing each word in a sequential manner . There is a strong inductive bias , though usually implicit , that the structure of a sentence is a sequence of words . However , natural language has complex underlying syntactic structures . Although recent work ( Kuncoro et al. , 2018 ) has demonstrated that RNNs are capable of capturing structural dependencies as long as they have enough capacity , in practice , it is less clear whether RNNs are able to capture such structural information with limited resources , especially over long distances . The underlying structures of sentences are useful in natural language understanding and generation . To fully understand the meaning of a sentence , a linguistic analysis not only needs to recognize the semantics of words in the sentence , but also needs to understand how the words are organized in the sentence . To compose a correct sentence , one should also follow grammars and syntactic rules . Unfortunately , conventional recurrent architectures are not specifically designed for modeling such structures . As a result , a poor performance can be expected for tasks requiring structural information . Traditionally , natural language parsers are used to model grammar and syntactic knowledge . By generating a parse tree , the parser can describe the structure and word dependencies of a sentence . However , to generate such a structure often involves some form of supervised training of a parser . Recent works have demonstrated several ways of learning structural information of natural sentences without supervision , which can be helpful when performing language modeling tasks ( Cheng et al. , 2016 ; Shen et al. , 2017 ) . In contrast to RNNs , convolutional neural network ( CNNs ) might be good candidates for learning structures due to their use of hierarchical filters . Recently , increasing attention has been paid to the use of CNNs for language modeling tasks . Dauphin et al . ( 2016 ) propose a gated convolutional language model , and the experiments in Bai et al . ( 2018 ) show that temporal convolutional networks ( TCNs ) perform almost as well as RNNs for many tasks . However , there is a noticeable performance gap between TCNs and RNNs . Furthermore , those models are not easily interpretable in that they do not explicitly learn the structures of sentences . A more dedicated architecture for modeling non-sequential structures is based on graph networks ( GNs ) ( Scarselli et al. , 2009 ) . The underlying structure among nodes is described with adjacency matrices on which the model propagates messages between different nodes ( Kipf & Welling , 2016 ; Li et al. , 2016 ) . In this work we aim to demonstrate how a non-recurrent neural model can more explicitly model structural information in sentences without supervision for improving language modeling tasks . Specifically , this non-recurrent neural model consists of a novel convolutional architecture combined with graph network blocks . To avoid the sequential inductive bias of natural sentences , we regard the underlying structures of the sentences as fully connected graphs to learn as much syntactic information as possible . We also apply a message propagation mechanism to learn the representations of words and sentences . Though individual modules used in this paper have been previously reported , to our knowledge , the combination of these components to achieve the reported results on language modeling tasks is novel . The contributions of this work are as follows : 1 . We show how RNNs model implicit connections between words , though they carry a sequential inductive bias . 2 . We propose a graph- and convolution- based language model , which can run in parallel . To our knowledge , the model achieves state-of-the-art results on the Penn Treebank dataset compared to existing non-recurrent language models . 3 . We introduce an unsupervised approach to learn syntactic parsing structure and other dependencies among words , which further improves the performance of our proposed language model . 2 RELATED WORK . RNNs have been widely applied to language modeling tasks for over a decade . For example , Cheng et al . ( 2016 ) use a long short-term memory ( LSTM ) RNN to learn a self-attention mechanism that indicates contextual dependencies in a language modeling task . As another example , Shen et al . ( 2017 ) propose an unsupervised learning method for syntactic distance , in which the language model can generate a parsing tree of a sentence . To achieve faster training speed , Gao et al . ( 2018 ) adopts a group strategy for recurrent layers . Also , inspired by human speed reading mechanisms , several RNN variants ( Seo et al. , 2017 ; Campos et al. , 2017 ) have been proposed to decide how to update their hidden states based on the importance of input tokens in a sequence . In contrast to RNN-based language models , it has been recently argued that temporal convolutional networks ( TCNs ) have competitive performance in some NLP tasks including word-level and character-level language modeling ( Bai et al. , 2018 ) . As is the case for RNNs , TCNs can not easily model relational information well by design . Many kinds of sequential data can be represented as graphs , in which nodes stand for entities and edges stand for relations . In order to model relational structures , our framework is based on graph neural networks ( GNNs ) , which were first proposed in ( Scarselli et al. , 2009 ) . Subsequently , the authors in Li et al . ( 2016 ) replaced the Almeida-Pineda algorithm used in ( Scarselli et al. , 2009 ) with backpropagation . In ( Garcia & Bruna , 2018 ) GNNs were applied to image classification tasks in a “ few-shot ” setting . In ( Gilmer et al. , 2017 ) , GNNs were applied to molecular property prediction tasks . The authors also summarized and generalized a message-passing mechanism . Kipf & Welling ( 2016 ) proposed graph convolution networks ( GCNs ) for semi-supervised learning , which conduct message propagation without learning edge representations . Li et al . ( 2016 ) introduce a gating mechanism , which improves the performance of GNNs . Unfortunately , there are relatively few papers discussing how to adapt GNNs to natural language tasks . Marcheggiani & Titov ( 2017 ) applied GNNs to semantic role labeling . Schlichtkrull et al . ( 2017 ) used GNNs to perform knowledge base completion tasks . ( Johnson , 2017 ) generated a graph based on textual input and update the relationship during training . Scarselli et al . ( 2009 ) proposed a neural model for message propagation in graphs , using an adjacency matrix , A , of a target graph to maintain structural information . Finally , ( Seo et al. , 2016 ) combined GNNs with RNNs for graph structure data . 3 BACKGROUND . 3.1 LONG SHORT-TERM MEMORY AND VARIANTS . LSTM ( Hochreiter & Schmidhuber , 1997 ) attempts to solve gradient problems associated with training recurrent networks and memorize longer sequences . The basic feedforward propagation method of LSTMs is shown as follows : zt = tanh ( W x zxt +W h zht−1 + b z ) ft = σ ( W x fxt +W h fht−1 + b f ) ot = σ ( W x oxt +W h oht−1 + b o ) it = σ ( W x i xt +W h i ht−1 + b i ) ct = ft ct−1 + it zt ht = ot tanh ( ct ) ( 1 ) A variant of LSTMs , named Quasi-RNNs ( QRNN ) ( Bradbury et al. , 2016 ) , calculates z , f , o , i in parallel by replacing ht−1 with xt−1 , which significantly accelerates training . However , QRNNs still have to calculate cell values sequentially . Applying the inductive bias that the underlying structure of natural sentences are linear sequences , LSTMs only explicitly model sequential structures . The relation between two words with a distance larger than 1 is modeled implicitly with a sequence of input and forget gates . Given equation 1 , we can rewrite the message propagation in LSTM cells in the form of c = A · z , where c = [ c1 , c2 , . . . , ct ] T , and z = [ z1 , z2 , . . . , zt ] T as follow : c = i1f2i1 i2· · · · · ·∏t i=2 fi · i1 ∏t i=3 fi · i2 · · · it ︸ ︷︷ ︸ A z1z2· · · zt ( 2 ) In the above equation , each element in A , Aij indicates the amount of message passed from zj to zi , where aij , i 6=j = i∏ j fj+1 · ij ( 3 ) aii = ii ( 4 ) Inspired by recent work on graphical neural models ( Scarselli et al. , 2009 ; Kipf & Welling , 2016 ; Veličković et al. , 2017 ) , we can regard A as the adjacency matrix of a weighted , directed graph G = ( z , E ) , where z = [ z1 , z2 , . . . , zt ] T are nodes and each edge e ∈ E indicates the amount of information needed to be propagated from one node to another . Equation 2 shows how LSTMs implicitly learn relation between each pair of words in a sentence . 3.2 GRAPH NEURAL NETWORKS . We argue that sentences can be structured as graphs , in which nodes stand for entities and edges stand for relations . Following ( Scarselli et al. , 2009 ; Kipf & Welling , 2016 ) , the message propagation in a GCN is given by : Xl+1 = f ( D− 1 2AD− 1 2 ·XlW ) ( 5 ) where xl and xl+1 are the input and output of the current layer , f is a non-linear activation , D is the diagonal degree matrix of adjacency matrix A , and W stands for a learnable weight matrix in the current layer . The gating mechanism can also be applied to message propagation in graphical models . Li et al . ( 2016 ) propose gated graph neural networks ( GGNNs ) , which filter the embeddings of the target node and its neighbors with gates instead of simply averaging them as in GCNs . 4 GRAPH TEMPORAL CONVOLUTIONAL NETWORKS . Inspired by the analysis of LSTMs and recent work on graphical neural models , in this section we introduce graph temporal convolutional networks ( GTCNs ) for language modeling . We are motivated by the fact that we can calculate the adjacency matrix , A , in parallel , and thus avoid the difficulties of training recurrent neural structures and adopting the sequential hypothesis for language models . The GTCN model consists of two modules : context attention and message propagation . 4.1 POSITION-AWARE CONTEXT ATTENTION . To decide how much the previous words contribute to predicting the next word , the GTCN model generates an attention over a context for each target word . Given the input sequence consists of n words , X = [ x1 , x2 , . . . , xn ] , we calculate key and value vectors , k and v , for each word for calculating the attention , qi = tanh ( W q 1 · xi +W q 2 · xi−1 + bq ) ( 6 ) ki = tanh ( W k 1 · xi +Wk2 · xi−1 + bk ) ( 7 ) where ∗ is a convolution operation . For words xi and xj ( j < i ) , we calculate their relation aij as follows , aij = eij∑ s < k < i eik ( 8 ) eij = exp ( ki · ( vj +Wpi−j ) + b e ) ( 9 ) where s is the start location of a visible window , and Wpd is the relative position representation proposed in ( Shaw et al. , 2018 ) , which encodes the distances between different words .

Summary: This paper draws inspiration from recent works on graph convolutional networks and proposes GTCN, a convolutional architecture for language modeling. The key intuition is to treat sentences as (potentially densely-connected) graphs over tokens, instead of sequences as in many RNN-based language models. The model then, when predicting a token, summarizes previous tokens using attention mechanism as context. Empirical evaluation on word-level language modeling on Penn Treebank shows competitive performance.