Transfer Active Learning For Graph Neural Networks

Graph neural networks have been proved very effective for a variety of prediction tasks on graphs such as node classification . Generally , a large number of labeled data are required to train these networks . However , in reality it could be very expensive to obtain a large number of labeled data on large-scale graphs . In this paper , we studied active learning for graph neural networks , i.e. , how to effectively label the nodes on a graph for training graph neural networks . We formulated the problem as a sequential decision process , which sequentially label informative nodes , and trained a policy network to maximize the performance of graph neural networks for a specific task . Moreover , we also studied how to learn a universal policy for labeling nodes on graphs with multiple training graphs and then transfer the learned policy to unseen graphs . Experimental results on both settings of a single graph and multiple training graphs ( transfer learning setting ) prove the effectiveness of our proposed approaches over many competitive baselines . 1 INTRODUCTION . Graphs encode the relations between different objects and are ubiquitous in real-world . Learning effective representations of graphs is critical to a variety of applications . Recently , graph neural networks ( Kipf & Welling , 2016 ; Veličković et al. , 2017 ) have been attracting growing attention , which iteratively update node representations by combining information of the nodes and their neighbors . They have been proved effective in many applications such as node classification ( Kipf & Welling , 2016 ; Veličković et al. , 2017 ) and link prediction ( Zhang & Chen , 2018 ; Cao et al. , 2018 ) . Despite the appealing performance , the training of graph neural networks typically requires massive labeled data ( Yang et al. , 2016 ) . However , in many domains such as healthcare , a large number of labeled data may not be available , and it could be very expensive to obtain labeled data . Therefore , in this paper we focus on how to minimize human efforts in obtaining labeled data . Specifically , we study active learning for graph neural networks . In other words , our goal is to select informative nodes on graphs and query for their labels to train graph neural networks . There are already some existing work that studied active learning for graphs ( Gadde et al. , 2014 ; Ji & Han , 2012 ; Cai et al. , 2017 ) , which generally used some optimization theory or proposed different kinds of heuristics ( e.g. , using the degree of nodes or the entropy of predicted label distributions ) to select informative nodes for labeling . However , the nodes selected by these heuristics may not be optimal for a specific task . Moreover , these rules are usually domain specific , which are difficult to generalize to new graphs . In this paper , we formulate active learning on graphs as a sequential decision process and aim to learn a policy network to sequentially pick the informative nodes for optimizing a specific task . Taking the node classification as an example , we define the state as the graph with the node information as whether the nodes are selected or not and their label distributions predicted by current graph neural networks . The action is picking the next node for labeling . After a node is picked , we retrain the graph neural network by adding the new picked node into the training set . The performance gain on the validation data set is defined as the reward . A policy network , which encodes the state with another graph neural network and predicts the next node to label , is trained to optimize the performance of graph neural network for the specific task . The above approach allows us to learn a policy to select informative labeled nodes for training graph neural networks on a single graph . However , a more important question is whether the learned policy can be transferred to new unseen graphs . Therefore , we further study transfer active learning for graph neural network , which aims to learn a universal policy with multiple training graphs to generalize to new graphs . Given multiple training graphs , a straightforward solution is to learn a single policy across multiple training graphs . Such an approach works reasonable well in practice . However , in some cases , the optimal policy on different graphs may be different from each other . Therefore , we allow each graph to have a graph-specific policy , which is an instantiation of the universal policy on a different graph . During training , each graph-specific policy is trained to maximize the performance of its own task and meanwhile regularized by the universal policy , and the knowledge learned by different graph-specific policies is dynamically distilled into the universal policy by minimizing the KL divergence between the action distributions . With such a strategy , the universal policy can acquire knowledge from multiple graphs , and is therefore capable of transferring to new graphs . We evaluate our proposed methods on the standard semi-supervised node classification task , in which the policy is trained and evaluated on a single graph , and the transfer active learning setting , in which the universal policy is trained with multiple graphs and evaluated on new graphs . Experimental results on both settings prove the effectiveness of our proposed approaches over many competitive baselines . 2 RELATED WORK . There are some existing work on active learning for graphs . Early studies ( Guillory & Bilmes , 2009 ; Ji & Han , 2012 ; Gadde et al. , 2014 ) typically formalized the task as an optimization problem . There are also works on semi-supervised classification on graphs with very few labeled data , such as Gallagher et al . ( 2008 ) ; Lin & Cohen ( 2010 ) . Nevertheless , these methods are very computational expensive and can not scale up to real-world graphs . Some recent studies ( Cai et al. , 2017 ) approached active learning by linearly combining several hand-crafted features including the uncertainty of the predictions , structure features such as centrality , and semantic features . More recently , Abel & Louzoun ( 2019 ) proposed to take the diversity of a node ’ s neighbors in terms of label into consideration and choose the node to be labeled in a more regional perspective . However , the labeled nodes selected by these heuristics may not be optimal for the specific task while our policy network is trained to optimize the performance of the specific task . Moreover , these heuristics could be difficult to generalize to new unseen graphs . Our work is also related to learning to learn strategies for active learning . Fang et al . ( 2017 ) studied using reinforcement learning for active learning on the task of Named Entity Recognition ( NER ) , where they need to decide which sentence should be queried to label the Named Entity inside it . They formalized the sentence selection as a sequential decision process to sequentially select the sentences to be labeled . Similarly , Liu et al . ( 2018a ) used imitation learning to train an active learning policy for NER tasks . Liu et al . ( 2018b ) applied reinforcement learning algorithm to learn the policy for active learning on Neural Machine Translation tasks . Bachman et al . ( 2017 ) proposed to learn the policy network for active learning via meta-Learning . Different from these works , in this paper we focus on active learning for graphs , in which the data ( the nodes ) to be labeled are highly correlated , and hence the problem is more challenging . Moreover , our goal is learn a unified policy for active learning to transfer across different domains or graphs . 3 METHODOLOGY . 3.1 PROBLEM DEFINITION . Graph neural networks have been proved effective in a variety of predictive tasks on graphs but require many labeled data for training . In this paper , we focus on how to minimize humans ’ efforts in obtaining labeled data on graphs ( i.e . active learning ) for training graph neural networks . Take the node classification as an example . We formulate the problem as a sequential decision process and aim to learn a policy to select the most informative nodes and further query for their labels for training graph neural networks . Formally , for a graph G = ( VG , EG , FG , LG ) , where VG , EG , FG and LG stand for nodes , edges , node features and node labels respectively , we aim at learning a policy πG , which selects a sequence of nodes V ′G ⊆ VG from the training set with the size of V ′G controlled by a given budget . We expect that the graph neural network trained with the labels of V ′G has the optimal prediction performance on test sets of G. Although active learning on single graphs is important , in practice we care more about whether the learned policy can transfer to unseen graphs . Therefore , we further propose to study transfer active learning for graph neural networks , where a universal policy is learned on multiple training graphs and evaluated on unseen test graphs . Formally , given a set of training graphs Strain , we seek to learn a universal policy πu for selecting informative labeled nodes on graphs , which is able to transfer to new unseen graphs in the test graph set G ∈ Stest . We first introduce our approach for learning a policy network on a single graph and then introduce how to learn a universal policy with multiple training graphs . 3.2 ACTIVE LEARNING ON A SINGLE GRAPH . Recall that our goal is to learn a policy for selecting a set of nodes for annotation , such that a graph neural network trained with the labels of the selected nodes can have the optimal performance on test nodes . We formalize the problem as a sequential decision process ( a.k.a. , Markov decision process . We define the state as the current graph with the node attributes as whether they are picked or not and their label distributions predicted by the current graph neural network , and the action as picking a node for which we query the label . Once a node is picked , we will add this node into the training set and retrain the graph neural network , which will update the state . The performance gain on validation set is treated as the intermediate reward . The goal is to maximize the total sum of the reward , i.e. , the final performance of the graph neural network . More specifically , for the state representation , we define the node features as follows : • Heuristic Features . One effective criterion to active learning is the uncertainty of the prediction given by the training model , i.e. , the entropy of the label distribution computed by the current graph neural network . Moreover , we also use the degree of a node , which is widely used in existing graph active learning methods ( Cai et al. , 2017 ) . The details of heuristic features are presented in Appendix . Note that all the heuristics we use are transferable to other unseen graphs . • Structural Features . Another observation is that nodes with similar structural roles should be similarly informative . Therefore , we also use some structural features . Specifically , we use the Struc2vec model ( Figueiredo et al. , 2017 ) to learn a latent representation for each node , which captures the local role of nodes . Struc2vec model allows joint training over different graphs , so once we learn the latent representations for all nodes of the graph that we are interested in , transfer learning can be done on this unified semantic space . • Historical Features . Another important information is the historical information , i.e. , which nodes are already annotated and which nodes are not . Motivated by that , we add an extra dimension in the feature vector of a node , where 1 means the node is already annotated , and 0 otherwise . Moreover , we also average the struc2vec features of all previously annotated nodes to capture the historical information . After getting the feature vector xsv defined above for each node v at the state s , we pass them to a 2-layer graph convolutional networks to propagate the information of a node to its neighbors , and the final node representation , i.e. , the state representation , is computed as zsv = GCN ( x s v ) . We solve the problem by training a policy network π , which defines a distribution over actions conditioned on states , i.e. , the probability of selecting a node given the current state π ( a|s ) . We parameterize the policy network with a linear energy-based model as follows : π ( a = v|s ) = 1 Z exp ( wT zsv ) , ( 1 ) where w is a weight vector , and Z is the normalization term . Our goal is to maximize the total sum of reward following the policy π , which can be characterized by the following objective function : OG ( π ) = Es∼dπ ( s ) Ea∼π ( a|s ) [ rG ( s , a ) ] , ( 2 ) where dπ ( s ) = ∑ k=0 γ kPr ( s0 → s , k , π ) is the stationary state distribution on graph G induced by the policy π , with Pr ( s0 → s , k , π ) being the probability that the agent arrives at state s in the k-th step starting from state s0 , and γ being the decay factor . rG ( s , a ) is the reward function on that graph , which is computed as the performance gain on the validation set after taking action a at s. Such an objective function can be effectively optimized by using the REINFORCE algorithm ( Williams , 1992 ) , where we sample some trajectories from the policy , and further update the parameters . Next , we introduce how we learn a unified policy in the transfer learning setting .

Summary: In this paper, the authors proposed a new method for active learning on node classification with GCN. RL based framework is used. The labeled graph is treated as state and the action is labeling the nodes. Validation accuracy on the hold out set is used as reward. Further transfer learning framework is also proposed, where graph-specific policy and master policy are jointly learned. Experiments on benchmark dataset show the effectiveness of proposed method compared to several baselines.