Dual Graph Complementary Network

As a powerful representation learning method on graph data , graph neural networks ( GNNs ) have shown great popularity in tackling graph analytic problems . Although many attempts have been made in literatures to find strategies about extracting better embedding of the target nodes , few of them consider this issue from a comprehensive perspective . Most of current GNNs usually employ some single method which can commendably extract a certain kind of feature but some equally important features are often ignored . In this paper , we develop a novel dual graph complementary network ( DGCN ) to learn representation complementarily . We use two different branches , and inputs of the two branches are the same , which are composed of structure and feature information . At the same time , there is also a complementary relationship between the two branches . Beyond that , our extensive experiments show that DGCN outperforms state-of-the-art methods on five public benchmark datasets . 1 INTRODUCTION . Although many attempts have been made in literatures to find a better strategy to learn the target node representation , the feature extraction capabilities of most methods are still far from optimal , especially when only a small amount of data is labeled . However , in fact , compared with the expensive and laborious acquisition of labeled data , unlabeled data is much easier to obtain . Therefore , how to learn more useful representations with limited label information is the key direct of representation learning study . Methods of this issue , commonly referred to as semi-supervised learning , which essentially believe that the similar points have similar outputs . Thus , it can properly utilize the consistency of data to make full use of the rich information of unsupervised data . In the real world , it is common that we have data with specific topological structures which usually called graph data . The graph structure is usually expressed as the connection between nodes . By aggregating the features of neighborhood and performing appropriate linear transformation , graph neural networks ( GNNs ) can convert graph data into a low-dimensional , compact , and continuous feature space . Nevertheless , most of them only care about a single aggregation strategy , which is counter intuitive : for example , as far as social networks are concerned , the relationship between people is very complex , while , most of the traditional GNNs only consider the single connection between nodes and ignore other implicit information . In this paper , our work focuses on learning node representations by GNNs in a semi-supervised way . Despite there are already many graph-based semi-supervised learning methods ( Kipf & Welling , 2016 ; Yang et al. , 2016 ; Khan & Blumenstock , 2019 ) , most of them can only find a single relationship between nodes . As a result , some information in unsupervised data is usually ignored . To overcome this problem , we develop a novel dual graph complementary network ( DGCN ) to extract information from both feature and topology spaces . An intuition of our method is to learn based on disagreement : network performance is largely related to the quality of the graph , which usually emphasizes the relevance of an attribute of instances . So , since we don ’ t know what attributes are most important , we consider both of them in the model design . Compared with the traditional GNN-based methods , we perform two different aggregate strategies which emphasize different attributes in each branch , one from the perspective of node feature , and the other from the topological structure . Then , to further utilize implicit information , we employ two networks with different structures to extract embedding from input feature . By doing so , nodes ’ information can be propagated in different ways . Then , the supervised loss ℓsup and diversity constraint ℓdiv are used to guide the training . We use two different branches to extract common information in topology and feature spaces . By utilizing disagreements between the two branches , model can gain information that may be ignored by single branch . To prove the effectiveness of our method , we conducted experiments on five public benchmark datasets . The contributions of our work are summarized as follows : • We propose a novel dual graph complementary network ( DGCN ) to fuse complementary information , which utilizes different graphs to aggregate nodes that are similar in certain attributes in a complementary way . • By comparing with algorithms that use non-single graphs , it proves that our complementary architecture can extract richer information • Through extensive evaluation on multiple datasets , we demonstrate DGCN effectiveness over state-of-the-art baselines . 2 RELATED WORK . 2.1 SEMI-SUPERVISED LEARNING . Semi-supervised learning is usually aimed at the case of insufficient data labels . X ∈ Rn×d is the feature of input nodes . Y = [ yij ] ∈ Rn×k is the label matrix , where k is the class number . yij means that the i-th node belongs to the j-th class . Then split data points into labeled and unlabeled points . Accordingly , xL and xU express a feature of labeled and unlabeled instance , respectively . Moreover , the ground-truth label of the label nodes is available only . The main objective of semi-supervised learning is to extract supervised information from labeled dataset whilst adequately utilizing data distribution information contained in X . There are four categories of semi-supervised learning algorithms : 1 . Self-training semi-supervised learning ( Lee , 2013 ) : It utilizes high-confidence pseudo labels to expand label set . Ideally , it can continuously improve network performance , but is usually limited by the quality of pseudo labels . 2 . Graph-based semi-supervised learning : It propagates information between instances according to edges in graph . It ’ s an inductive learning method , of which the performance mainly depends on the aggregation algorithm . 3 . Low-density separation methods ( Joachims , 1999 ) : They assume that the decision hyperplane is consistent with the data distribution , so so it should pass through the sparse region of the data . 4 . Pretrain semi-supervised learning : such as autoencoder ( Vincent et al. , 2008 ; Rifai et al. , 2011 ) , trains the model based on reconstruction error and then fine tune it using labeled data . However , semi-supervised learning tasks prefer to obtain information related to data distribution rather than all information of samples . In this paper , we mainly focus on the graph-based semisupervised learning . 2.2 GRAPH-BASED SEMI-SUPERVISED LEARNING . In addition to features , graph-based semi-supervised learning methods ( Kipf & Welling , 2016 ) represent the topological edge connection between different instances . For many datasets , graph is given as a feature . If the features of the dataset do not contain the relationships between different samples , a graph can also be constructed by measuring the similarity between the features of the instances ( Zhu et al. , 2003 ) . Actually , the graph is a measure of whether the instances are closely connected . Then , according to this graph , information exchange between instances can be carried out , so that the information of unlabeled data can be effectively utilized . Network performance is largely related to the quality of the graph . When the attributes emphasized in the graph do not match the expectations of the task objective , misjudgments are often caused . Usually , it is difficult to finding what really matters . The traditional graph-based semi-supervised learning methods usually uses a single graph for node aggregation , which causes a single attribute to be emphatically considered , but when this attribute does not match the task goal , it will mislead the training instead . 3 DGCN ARCHITECTURE . In this section , we will present the overall framework of DGCN , see Fig . 1 . The main idea of DGCN is that information exchange under the control of graphs emphasizing different attributes can extract more abundant features . To this end , we use two branches to extract information from two inputs at the same time . The node features of these two inputs are the same , the only difference is the graphs that control the information exchange . In addition , in order to further expand the difference between branches , we use a diversity loss ℓdiv . 2 , l , H gat 1 , l and Hgat2 , l respectively . Then , we fuse GCN view and GAT view respectively to obtain H gcn c and H gat c respectively through attention operation . The obtained Hgcnc and H gat c are sent to the final attention layer together with the previous Hgcn1 , l , H gcn 2 , l , H gat 1 , l and H gat 2 , l . 3.1 NOTATION & PROBLEM STATEMENT . Let G = ( V , A , X ) be an undirected graph . V is the set of nodes on the graph , which is composed of unlabeled ( Vu ) and labeled ( Vl ) nodes with the number of nodes is nu and nl respectively . n = nl + nu is the number of nodes . A = [ aij ] ∈ Rn×n is the adjacency matrix . aij = 1 represents that node i and node j are closely related in an attribute , otherwise , aij = 0 . 3.2 BRANCHES . In order to capture different characteristics by the two branches ( also called viewer ) , we use different network structures for each branch : GCN ( Kipf & Welling , 2016 ) and GAT ( Veličković et al. , 2017 ) . Given a graph G = ( V , A , X ) , both GCN and GAT intend to extract richer features at a vertex by aggregating features of vertices from its neighborhood ( Li et al. , 2019 ) . So the node representation of the l-th layer Hl can be defined by : Hl = Update ( Aggregate ( Hl−1 , Θ agg l ) , Θ update l ) . ( 1 ) where Θaggl and Θ update l are the learnable weights of aggregation and update functions of the l-th layer respectively and the initial H0 = X . The aggregation and update functions are the essential components of GNNs , and obviously the features extracted by different aggregation functions will have certain differences . Thus , we take advantage of two different networks , GCN and GAT , to obtain node representation . The node features output by the l-th GCN layer can be expressed as : Hl = σ ( ( D̃− 1 2 ( A+ I ) D̃− 1 2 ) Hl−1Wl ) . ( 2 ) where I ∈ Rn×n indicates the identity matrix , A+ I means adding self-loop in the graph , D̃ is the diagonal degree matrix of A+ I , and σ ( · ) is the activation function . It can be seen from equation 2 that GCN aggregates neighbor features by weighting the value of symmetric normalized laplacian . Next , we introduce the algorithm GAT that uses the attention mechanism to calculate the neighbor weight . Through a learnable coefficient a , GAT can assign learnable weights to each neighbor of the node . For node i , the weight αij between it and its neighbor node j can be expressed as : αij = exp ( LeakyReLU ( a⊤ [ Whi∥Whj ] ) ) ∑ k∈Ni exp ( LeakyReLU ( a⊤ [ Whi∥Whk ] ) ) . ( 3 ) where ·⊤ is the transposition operation and ∥ represents concatenation . Then the forward propagation process of node v in l-th layer can be represented as : hl , i = ∥Mm=1σ ∑ j∈Ni αml , ijW m l hl−1 , j . ( 4 ) where , hl , i is the embeding of node i in the l-th layer . M is the number of independent attention mechanisms . σ is activation function of GAT . αmij is the normalized attention coefficients computed by the m-th attention mechanism , see equation 3 . As can be seen from equation 4 , the weights GAT assigns to a node ’ s neighbors are learnable . Thus we can assign adaptive weights to different neighbors . Although these two methods are based on the existence of connection between points as the premise of aggregation . Both the GCN and GAT models we use have their own advantages and disadvantages . The former considers the relationship between nodes ( probability conduction matrix ) , but can ’ t learn neighbor weights dynamically . Although the latter can assign dynamic weights to neighbors , it ignores the influence of degree attribute of node on aggregation . Therefore , using these two branches , we can extract more complementary features from the input .

Summary: This paper introduces a method on semi-supervised graph classification. For each graph, the method first constructs another view based on the cosine similarity between nodes' features, and from the two views (topology and feature similarity), GCN and GAT are applied to extract representations. All node representations are further combined via two layers of attentions. A diversity loss that encourages dissimilarity between the learned representations of GCN and GAT is introduced to the cross-entropy loss for joint optimization. The whole framework makes sense in terms of learning meaningful node representations for classification. However, the method lacks novelty, it is an incremental development on the existing graph neural networks. The choice of GCN and GAT as the building blocks are not well justified. It is also possible to try other kinds of GNNs. The statement on GAT "ignores the inherent structure of the graph space" on page 4 is confusing since it learns weights based on the graph structures. The experimental results show the better performance of the proposed method, but are not well analyzed. It may be better to compare with other multi-graph methods such as