Scalable Generative Models for Graphs with Graph Attention Mechanism

1 INTRODUCTION . Graphs are ubiquitous and fundamental data structures in the real world , appearing in various fields , such as social science , chemistry , and biology . Moreover , in applications such as drug discovery and network simulation , graph generative models that can approximate distributions over graphs on a specific domain and derive new samples from them are very important . Compared with generation tasks for images or natural language , graph generation tasks are significantly more difficult ; this is due to the necessity of modeling complex local/global dependencies among nodes and edges as well as the intractable properties of graphs themselves , such as discreteness , variable number of nodes and edges , and uncertainty of node ordering . Among the several challenges involved in graph generation tasks described above , scalability is one of the most important for applications in a wide range of real-world domains . In this paper , we define scalability from three perspectives : graph scalability , data scalability , and label scalability . Graph scalability denotes scalability to large graphs with many nodes within the limit of practical time/space complexity , especially in training . Data scalability denotes scalability to large datasets containing many data . Label scalability denotes scalability to graphs with many node/edge labels . Besides these , it is possible to consider other viewpoints , such as edge number , graph diameter , and the total number of nodes in a dataset . However , because these are closely related to the above three defined perspectives , we consider them as a mutually exclusive and collectively exhaustive division . In recent years , an increasing number of graph generative models based on machine learning have been proposed . They have demonstrated great performances in several tasks , such as link prediction , molecular property optimization , and network structure optimization ( Li et al. , 2018 ; You et al. , 2018b ; Grover et al. , 2018 ; Luo et al. , 2018 ; You et al. , 2018a ; Liu et al. , 2018 ; Simonovsky & Komodakis , 2018 ; Wang et al. , 2017 ; Li et al. , 2018 ) . However , to the best of our knowledge , no proposed model is scalable in all three contexts . For example , DeepGMG ( Li et al. , 2018 ) can generate only small graphs , and GraphRNN ( You et al. , 2018b ) can not consider node/edge labels . Besides , both models have weak compatibility with parallel training , which is key to efficient training on a large dataset . In this work , we propose the Graph Generative Model with Graph Attention Mechanism ( GRAM ) for generating graphs that is scalable in all three contexts , especially during training . Given a set of graphs , our model approximates their distribution in an unsupervised manner . To achieve graph scalability , we employ an autoregressive sequential generation process that is flexible to variable nodes and edges , and formulate the likelihood of graphs in a simple manner to simplify the generation process . Besides , we apply two techniques to reduce computational cost : breadth-first search ( BFS ) and zeroing attention weights in edge estimation . Regarding data scalability , we apply a novel graph attention mechanism that simply extends the attention mechanism used in natural language processing ( Vaswani et al. , 2017 ) to graphs . This graph attention mechanism does not include sequentially dependent hidden states as the recurrent neural network ( RNN ) does , which improves the parallelizability of training significantly . Compared with other graph attention mechanisms ( Veličković et al. , 2018 ; Abu-El-Haija et al. , 2018 ; Ishiguro et al. , 2019 ) , ours is architecturally simple , computationally lightweight , and general , i.e. , its applicability is not limited to generation tasks . Finally , for label scalability , we formulate the likelihood of graphs assuming multiple node/edge labels . Also , we use graph convolution/attention layers , whose numbers of parameters do not depend directly on the number of labels . Moreover , we introduce a unified non-domain-specific evaluation metric for generation tasks of node/edge-labeled graphs . Because such a method does not currently exist , prior works relied on domain-specific metrics or visual inspection , which made unified and objective evaluations difficult . Thus , we construct a unified evaluation metric that dispenses with domain-specific knowledge and considers not only the topology but also node/edge labels by combining a graph kernel and Maximum Mean Discrepancy ( MMD ) ( Gretton et al. , 2012 ) . Although a similar statistical test based on MMD via a graph kernel was used for schema matching of protein graphs in Kriegel et al . ( 2006 ) , to the best of our knowledge , this work is the first to use as an evaluation metric for graph generation tasks . Our experiments on synthetic and real-world graphs demonstrated that our models can scale up to handle large graphs and datasets that previous methods had difficulty with ; our models demonstrated superior results to those of baseline methods . To summarize , the contributions of this work are as follows : • We propose GRAM , a graph generative model that is scalable , especially in training . • We propose a novel graph attention mechanism that is general and architecturally simple as a key portion of the generative model . • We define scalability in graph generation from three perspectives : number of nodes , data , and labels . • We construct a unified non-domain-specific evaluation metric for node/edge-labeled graph generation tasks by combining a graph kernel and MMD . 2 RELATED WORK . Although there are several traditional graph generative models ( Erdös & Rényi , 1959 ; Albert & Barabási , 2002 ; Leskovec et al. , 2010 ; Robins et al. , 2007 ; Airoldi et al. , 2009 ) , we focus here on recent machine learning-based models , which have outperformed traditional models in various tasks . In terms of their generation process , existing graph generative models can be classified into at least two types : tensor generation models and sequential generation models . Tensor generation models ( Simonovsky & Komodakis , 2018 ; De Cao & Kipf , 2018 ; Grover et al. , 2018 ) generate a graph by outputting tensors that correspond to the graph . These models are architecturally simple and easy to optimize for small graphs . However , they face difficulty in generating large graphs owing to the non-unique correspondence between a graph and tensors , limitations in the pre-defined maximum number of nodes , and the increasing number of parameters depending on the maximum graph size . In contrast , sequential generation models , such as DeepGMG ( Li et al. , 2018 ) , GraphRNN ( You et al. , 2018b ) , generate graphs by adding nodes and edges one-by-one , which alleviates the above problems and generates larger graphs . To achieve graph scalability , we employ the latter . However , to generate a graph with n nodes and m edges , DeepGMG requires at least O ( mn2 ) operations because of its complex generation process . GraphRNN reduces this time complexity to O ( nM ) with a dataset-specific valueM by utilizing the BFS . However , it can not handle node/edge labels , mainly because it omits the feature extraction process , and relies mainly on the information in the hidden states of the RNN . Moreover , these models include sequentially dependent hidden states , which make training parallelization difficult . In contrast , our model employs a graph attention mechanism without sequentially dependent hidden states . This improves the parallelizability of training significantly . Also , by applying two approximation methods , we reduce the complexity to almost a linear order of n while conducting a rich feature extraction . A comparison of our models ’ and the baselines ’ complexities is given in Table 2 in Section A.1 . As another division , there are unsupervised learning approaches and reinforcement learning approaches . Given samples of graphs , unsupervised learning models ( Li et al. , 2018 ; You et al. , 2018b ; Simonovsky & Komodakis , 2018 ; Grover et al. , 2018 ) approximate the distribution of them in an unsupervised manner . Reinforcement learning models ( Luo et al. , 2018 ; You et al. , 2018a ; De Cao & Kipf , 2018 ; Liu et al. , 2018 ; Li et al. , 2018 ) learn to generate graphs to maximize a given return , such as QED ( Bickerton et al. , 2012 ) . Although reinforcement learning approaches have demonstrated promising results on several tasks , such as molecular generation and network architecture optimization , we employ an unsupervised approach ; this is because it is considered to be more advantageous when new samples that share certain properties with training samples are required or when the reward functions can not be designed ( e.g. , a family of pharmaceutical molecules against a certain disease is known , but the mechanism by which they work is not entirely known ) . 3 PROPOSED METHOD . An outline of this section is as follows : In Section 3.1 , we first describe notations and define the considered graph generation task . In Section 3.2 , we formulate the likelihood of graphs . In Section 3.3 , we present our proposed graph attention mechanism , which is the key portion of our model and is used in feature extraction and edge estimation . In Section 3.4 , we present our scalable graph generative model , GRAM . We describe two techniques to reduce the computational cost of edge estimation : BFS in Section 3.5.1 and zeroing attention weights in Section 3.5.2 . In Section 3.6 , we describe the training and propose an evaluation metric . 3.1 NOTATIONS AND PROBLEM DEFINITION . In this work , we define a graph G = ( V , E ) as a data structure that consists of a node set V = { v1 , ... , vn } and an edge set E = { ei , j |i , j ∈ { 1 , ... , n } } . For simplicity , we focus on undirected graphs . We assume nodes and edges are associated with node/edge labels and denote the number of them as a and b , respectively . For graph representation , we employ a tensor representation . Specifically , given node ordering π , we represent a pair of a graph and node ordering ( G , π ) as a pair of tensors ( Xπ , Aπ ) . Note that we assume the existence of node ordering in graphs , and π is a permutation function over { 1 , ... , n } . Xπ ∈ { 0 , 1 } n×a , whose i-th row is a one-hot vector corresponding to the label of vπ ( i ) , stores information about nodes . Aπ ∈ { 0 , 1 } n×n×b , whose ( i , j ) -th element is a one-hot vector corresponding to the label of eπ ( i ) , π ( j ) , stores information about edges . If no edge exists between vπ ( i ) and vπ ( j ) , we replace it with a zero vector . An example is illustrated in Figure 3 in Section A.2 . Note that ( G , π ) and ( Xπ , Aπ ) are in unique correspondence . Finally , for simplicity , we do not consider self-looping or multiple edges . A possible extension to self-looping edges would involve adding a step to estimate them . For multiple or directed edges , we can prepare additional labels that correspond to them . In the remainder of this paper , for clear notation , we assume π ( i ) = i and omit the superscript π . The target graph generation task is as follows : given samples from a distribution of graphs p ( G ) , approximate the distribution ( p̂ ≈ p ) such that we can derive new samples from it ( G ∼ p̂ ( G ) ) .

Summary: This paper presents new graph generative model (GRAM) which claims to tackle the scalability issue found is most of the published models. The propose architecture can scale on graphs from three perspective - number of graphs, number of nodes/edges and number of node/edge labels. This is achieved by sequentially generating the subgraph using graph attention and graph convolutional layers. While training, each of these subgraph can be trained in parallel. Further, paper applies couple of heuristics to reduce computational time and introduces a new non-domain specific evaluation metric for the generation of node/edge labeled graphs.