Unsupervised Universal Self-Attention Network for Graph Classification

1 INTRODUCTION . Many real-world and scientific data are represented in forms of graphs , e.g . data from knowledge graphs , recommender systems , social and citation networks as well as telecommunication and biological networks ( Battaglia et al. , 2018 ; Zhang et al. , 2018c ) . In general , a graph can be viewed as a network of nodes and edges , where nodes correspond to individual objects and edges encode relationships among those objects . For example , in online forum , each discussion thread can be constructed as a graph where nodes represent users and edges represent commenting activities between users ( Yanardag & Vishwanathan , 2015 ) . Early approaches focus on computing the similarities among graphs to build a graph kernel for graph classification ( Gärtner et al. , 2003 ; Kashima et al. , 2003 ; Borgwardt & Kriegel , 2005 ; Shervashidze et al. , 2009 ; Vishwanathan et al. , 2010 ; Shervashidze et al. , 2011 ; Yanardag & Vishwanathan , 2015 ; Narayanan et al. , 2017 ; Ivanov & Burnaev , 2018 ) . These graph kernel-based approaches treat each atomic substructure ( e.g. , subtree structure , random walk or shortest path ) as an individual feature , and count their frequencies to construct a numerical vector to represent the entire graph , hence they ignore node attributes , substructure similarities and global network properties . One recent notable strand is to learn low-dimensional continuous embeddings of the whole graphs ( Hamilton et al. , 2017b ; Zhang et al. , 2018a ; Zhou et al. , 2018 ) , and then use these learned embeddings to train a classifier to predict graph labels ( Wu et al. , 2019 ) . Advanced approaches in this direction have attempted to exploit graph neural network ( Scarselli et al. , 2009 ) , capsule network ( Sabour et al. , 2017 ) or graph convolutional neural network ( Kipf & Welling , 2017 ; Hamilton et al. , 2017a ) for supervised learning objectives ( Li et al. , 2016 ; Niepert et al. , 2016 ; Zhang et al. , 2018b ; Ying et al. , 2018 ; Verma & Zhang , 2018 ; Xu et al. , 2019 ; Xinyi & Chen , 2019 ; Maron et al. , 2019b ; Chen et al. , 2019 ) . These graph neural network ( GNN ) -based approaches usually consist of two common phases : the propagating phase and the readout phase . The former phase aims to iteratively update vector representation of each node by recursively aggregating representations of its neighbors , and then the latter phase applies a pooling function ( e.g. , mean , max or sum pooling ) on output node representations to produce an embedding of each entire graph ; and this graph embedding is used to predict the graph label . We find that these approaches are currently showing very promising performances , nonetheless the dependency aspect among nodes , which often exhibit strongly in many kinds of real-world networks , has not been exploited effectively due to lack of advanced computations within the propagating phase . Very recently , the universal self-attention network ( Dehghani et al. , 2019 ) has been shown to be very powerful in NLP tasks such as question answering , machine translation and language modeling . Inspired by this new attention network , we propose U2GNN – a novel universal self-attention graph neural network to learn plausible node and graph embeddings . Our intuition comes from an observation that the recurrent attention process in the universal self-attention network can memorize implicit dependencies between each node and its neighbors from previous iterations , which can be then aggregated to further capture the dependencies among substructures into latent representations in subsequent iterations ; this process , hence , can capture both local and global graph structures . Algorithmically , at each timestep , our proposed U2GNN iteratively exchanges a node representation with its neighborhood representations using a self-attention mechanism ( Vaswani et al. , 2017 ) followed by a recurrent transition to infer node embeddings . We finally take the sum of all learned node embeddings to obtain the embedding of the whole graph . Our main contributions are as follows : • In our proposed U2GNN , the novelty of memorizing the dependencies among nodes implies that U2GNN can explore the graph structure similarities locally and globally – an important feature that most of existing approaches are unable to do . • U2GNN can be seen as a general framework where we prove the powerfulness of our model in both supervised or unsupervised fashions . The experimental results on 9 well-known benchmark datasets for the graph classification task show that both our supervised and unsupervised U2GNN models produce new state-of-the-art ( SOTA ) accuracies in most of benchmark cases . • To our best of knowledge , our work is the first to show that a unsupervised model can noticeably outperform up-to-date supervised approaches by a large margin . Therefore , we suggest that future GNN works should pay more attention to the unsupervised fashion as well as not comparing supervised models with unsupervised models together . This is important in both industry and academic applications in reality where expanding unsupervised GNN models is more suitable due to the limited availability of class labels . 2 RELATED WORK . Early popular approaches are based on “ graph kernel ” which aims to recursively decompose each graph into “ atomic substructures ” ( e.g. , graphlets , subtree structures , random walks or shortest paths ) in order to measure the similarity between two graphs ( Gärtner et al. , 2003 ) . For this reason , we can view each atomic substructure as a word token and each graph as a text document , hence we represent a collection of graphs as a document-term matrix which describes the normalized frequency of terms in documents . Then , we can use a dot product to compute the similarities among graphs to derive a kernel matrix used to measure the classification performance using a kernel-based learning algorithm such as Support Vector Machines ( SVM ) ( Hofmann et al. , 2008 ) . We refer to an overview of the graph kernel-based approaches in ( Nikolentzos et al. , 2019 ; Kriege et al. , 2019 ) . Since the introduction of word embedding models i.e. , Word2Vec ( Mikolov et al. , 2013 ) and Doc2Vec ( Le & Mikolov , 2014 ) , there have been several efforts attempted to apply them for the graph classification task . Deep Graph Kernel ( DGK ) ( Yanardag & Vishwanathan , 2015 ) applies Word2Vec to learn embeddings of atomic substructures to create the kernel matrix . Graph2Vec ( Narayanan et al. , 2017 ) employs Doc2Vec to obtain embeddings of entire graphs in order to train a SVM classifier to perform classification . Anonymous Walk Embedding ( AWE ) ( Ivanov & Burnaev , 2018 ) maps random walks into “ anonymous walks ” which are considered as word tokens , and then utilizes Doc2Vec to achieve the graph embeddings to produce the kernel matrix . In parallel , another recent line of work has focused on using deep neural networks to perform the graph classification in a supervised manner . PATCHY-SAN ( Niepert et al. , 2016 ) adapts a graph labeling procedure to generate a fixed-length sequence of nodes from an input graph , and orders k-hop neighbors for each node in the generated sequence according to their graph labelings ; PATCHY-SAN then selects a fixed number of ordered neighbors for each node and applies a convolutional neural network to classify the input graph . MPNN ( Gilmer et al. , 2017 ) , DGCNN ( Zhang et al. , 2018b ) and DIFFPOOL ( Ying et al. , 2018 ) are end-to-end supervised models which share similar two-phase process by ( i ) using stacked multiple graph convolutional layers ( e.g. , GCN layer ( Kipf & Welling , 2017 ) or GraphSAGE layer ( Hamilton et al. , 2017a ) ) to aggregate node feature vectors , and ( ii ) applying a graph-level pooling layer ( e.g. , mean , max or sum pooling , sort pooling or differentiable pooling ) to obtain the graph embeddings which are then fed to a fully-connected layer followed by a softmax layer to predict the graph labels . Graph neural networks ( GNNs ) ( Scarselli et al. , 2009 ) aim to iteratively update the vector representation of each node by recursively propagating the representations of its neighbors using a recurrent function until convergence . The recurrent function can be a neural network e.g. , gated recurrent unit ( GRU ) ( Li et al. , 2016 ) , or multi-layer perceptron ( MLP ) ( Xu et al. , 2019 ) . Note that both stacked GCN and GraphSAGE multiple layers can be seen as variants of the recurrent function in GNNs . Other graph embedding models are briefly summarized in ( Zhou et al. , 2018 ; Zhang et al. , 2018c ; Wu et al. , 2019 ) . 3 THE PROPOSED U2GNN . In this section , we detail how to construct our U2GNN and then present how U2GNN learns model parameters to produce node and graph embeddings . Graph classification . Given a set of graphs { G1 , G2 , ... , GN } ⊆ G and their corresponding class labels { y1 , y2 , ... , yN } ⊆ Y , our U2GNN aims to learn a plausible embedding oG of each entire graph G in order to predict its label y . Each graph G is defined as G = ( V , E , X ) , where V is a set of nodes , E is a set of edges , and X ∈ R|V|×d represents feature vectors of nodes . In U2GNN , as illustrated in Figure 1 , we use a universal self-attention network ( Dehghani et al. , 2019 ) to learn a node embedding ov of each node v ∈ V , and then oG is simply returned by summing all learned node embeddings as follows:1 oG = SUM ( { ov , ∀v ∈ V } ) ( 1 ) Constructing U2GNN . Formally , given an input graph G , we uniformly sample a set ofN neighbors for each v ∈ V , and then use node v and its neighbors for the U2GNN learning process.2 For example , as illustrated in Figure 1 , we generate a set of N = 3 neighbors { 1 , 2 , 4 } for node 3 , and then consider { 3 , 1 , 2 , 4 } as an input to U2GNN where we leverage on the universal self-attention network ( Dehghani et al. , 2019 ) to learn an effective embedding of node 3 . Intuitively , the universal self-attention network can help to better aggregate feature vectors from neighbors of a given node to produce its plausible embedding . In particular , each node and its neighbors is transformed into a sequence of feature vectors which are then iteratively refined at each timestep – using a self-attention mechanism ( Vaswani et al. , 2017 ) followed by a recurrent transition ( TRANS ) with adding residual connection ( He et al. , 2016 ) and layer normalization ( LNORM ) ( Ba et al. , 2016 ) . Given a node v ∈ V , we consider a sequence of ( N +1 ) nodes { vi } N+1i=1 where v1 = v and { vi } N+1 i=2 are N sampled neighbors of v. We obtain an input sequence of feature vectors { h0vi } N+1 i=1 for which 1The experimental results in ( Xu et al. , 2019 ) show that the sum pooling performs better than the mean and max poolings . 2We sample a different set of neighbors at each training step . h0vi = Xvi ∈ R d. In U2GNN , at each step t , we feed { ht−1vi } N+1 i=1 as an input sequence and produce an output sequence { htvi } N+1 i=1 for which h t vi ∈ R d as follows : htvi = LNORM ( xtvi + TRANS ( xtvi ) ) ( 2 ) with xtvi = LNORM ( ht−1vi + ATT ( ht−1vi ) ) ( 3 ) where TRANS ( . ) and ATT ( . ) denote a feed-forward network and a self-attention network respectively as follows : TRANS ( xtvi ) = W2ReLU ( W1xtvi + b1 ) + b2 ( 4 ) where W1 ∈ Rk×d and W2 ∈ Rd×k are weight matrices , and b1 and b2 are bias parameters , and : ATT ( ht−1vi ) = N+1∑ j=1 αi , j ( V ht−1vj ) ( 5 ) where V ∈ Rd×d is a value-projection weight matrix ; αi , j is an attention weight which is computed using the softmax function over scaled dot products between the i-th and j-th input nodes : αi , j = softmax ( Qht−1vi ) · ( Kht−1vj ) √ d ( 6 ) where Q ∈ Rd×d and K ∈ Rd×d are query-projection and key-projection matrices , respectively.3 Algorithm 1 : The unsupervised learning process . 1 Input : G = ( V , E , X ) 2 for v ∈ V do 3 SAMPLE { vi } N+1i=1 where { vi } N+1 i=2 are N sampled neighbors of v1 = v 4 for t = 1 , 2 , ... , T do 5 ∀i ∈ { 1 , 2 , ... , ( N + 1 ) } 6 xtvi ← LNORM ( ht−1vi + ATT ( ht−1vi ) ) 7 htvi ← LNORM ( xtvi + TRANS ( xtvi ) ) 8 ov ← hTv , ∀v ∈ V 9 oG = SUM ( { ov , ∀v ∈ V } ) Algorithm 2 : The supervised learning process . 1 Input : G = ( V , E , X ) with its label y 2 for v ∈ V do 3 SAMPLE { vi } N+1i=1 where { vi } N+1 i=2 are N sampled neighbors of v1 = v 4 for t = 1 , 2 , ... , T do 5 ∀i ∈ { 1 , 2 , ... , ( N + 1 ) } 6 xtvi ← LNORM ( ht−1vi + ATT ( ht−1vi ) ) 7 htvi ← LNORM ( xtvi + TRANS ( xtvi ) ) 8 oG = SUM ( { hTv , ∀v ∈ V } ) 9 y←WoG + b Learning parameters of U2GNN in “ unsupervised ” fashion : After T steps , we use the vector representation hTv1 to infer node embeddings ov1 . For example , as shown in Figure 1 , we have v1 = 3 , v2 = 1 , v3 = 2 and v4 = 4 , and then consider hT3 to infer o3 . We learn our model parameters ( including the weight matrices and biases as well as node embeddings ov ) by minimizing the sampled softmax loss function ( Jean et al. , 2015 ) applied to node v ∈ V as follows : LU2GNN ( v ) = − log exp ( ov · hTv ) ∑ v′∈V′ exp ( ov′ · h T v ) ( 7 ) where V ′ is a subset sampled from V . We briefly describe the general learning process of our proposed U2GNN model in Algorithm 1 . Here , the learned node embeddings ov are used as the final representations of nodes v ∈ V . After that , we obtain the plausible embedding oG of the graph G by summing all learned node embeddings as mentioned in Equation 1 . Finally , we employ a logistic regression classifier from LIBLINEAR ( Fan et al. , 2008 ) to predict the graph labels . 3It is important to note that GAT ( Veličković et al. , 2018 ) borrows the standard attention technique from ( Bahdanau et al. , 2015 ) in using a single-layer feedforward neural network parametrized by a weight vector and then applying the LeakyReLU non-linearity function followed by the softmax function to compute importance weights of neighbors of a given node . Therefore , GAT is much different with the self-attention mechanism . More information about GCN , GraphSAGE and GAT can be found in Appendix A . Learning parameters of U2GNN in “ supervised ” fashion : We do not need to learn the node embeddings separately , hence after T steps , we use hTv1 as the final embedding of node v1 ∈ V ( i.e. , ov = hTv ) . We then feed oG to a fully connected layer to predict the graph labels as briefly presented in Algorithm 2 . Finally , we learn the model parameters by minimizing the cross entropy loss function . Proof : In general , on the node level , each node and its neighbors are iteratively attended in the recurrent process with weight matrices shared across timesteps and iterations , thus U2GNN can memorize the potential dependencies among nodes within substructures . On the graph level , U2GNN views the shared weight matrices as memories to access the updated node-level information from previous iterations to further aggregate broader dependencies among substructures into implicit graph representations in subsequent iterations . Therefore , U2GNN is advantageous to capture both global and local graph structures to learn effective node and graph embeddings , leading to state-of-the-art performances for the graph classification task .

Summary: In this paper, the authors developed a graph embedding method called U2GAN based on self-attention mechanism. Similar to many existing graph neural network, U2GAN samples and aggregate neighboring features for each node in a graph. The aggregation function is similar to GAT, i.e., a query based attention layer. The difference is an incorporation of a transition function followed the attention layer. By minimizing Eq. 7, node embeddings can be inferred, which are summed up to obtain a graph embedding, for the downstream graph classification task.