Towards Robust Graph Neural Networks against Label Noise

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have achieved great success in various domains , but the necessity of collecting large amount of samples with high-quality labels is both expensive and time-consuming . To address this problem , cheaper alternatives have emerged . For example , the onerous labeling process can be completed on some crowdsourced system like Amazon Mechanical Turk 1 . Besides , we can collect labeled samples from web with search engines and social media . However , all these methods are prone to produce noisy labels of low quality . As is shown in recent research ( Zhang et al. , 2016b ) , an intractable problem is that DNNs can easily overfit to noisy labels , which dramatically degrades the generalization performance . Therefore , it is necessary and urgent to design some valid methods for solving this problem . Graph Neural Networks ( GNNs ) have aroused keen research interest in recent years , which resulted in rapid progress in graph-structured data analysis ( Kipf & Welling , 2016 ; Velickovic et al. , 2017 ; Xu et al. , 2018 ; Hou et al. , 2019 ; Wang & Leskovec , 2020 ) . Graph node classification is the mostcommon issue in GNNs . However , almost all the previous works about label noise focus on image classification problem and handling noisy labels in the task of graph node classification with GNNs has not been studied yet . Fortunately , most edges in the graph-structured datasets are intra-class edges ( Wang & Leskovec , 2020 ) , indicating that a node ’ s label can be estimated by its neighbor nodes ’ labels . In this paper , we utilize this special attribute of graph data to alleviate the damages caused by noisy labels . Moreover , meta learning paradigm serves as a useful tool for us to learn a proper aggregation between origin labels and pseudo labels as the final labels . The key contributions of this paper are as follows : • To the best of our knowledge , we are the first to focus on the label noise existing in utilizing GNNs to classify graph nodes , which may serve as a beginning for future research towards robust GNNs against label noise . • We utilize meta-learning to learn how to aggregate origin labels and pseudo labels properly to get more credible supervision instead of learning to re-weight different samples . 1https : //www.mturk.com/ We experimentally show that our LPM outperforms state-of-the-art algorithms in utilizing GNNs to classify graph nodes with both synthetic and real-world label noise . 2 RELATED WORK . 2.1 GRAPH NEURAL NETWORKS . To start , we use G = ( V , E , X ) to denote a graph whose nodes set is V and edges set is E , and X ∈ Rn×d is the input feature matrix , where n denotes the number of nodes in the graph and d is the dimension of the input feature vector of each node . We use eu , v ∈ E to denote the edge that connects node u and v. For each node v ∈ V , its neighbor nodes set can be donated as Nv = { u : eu , v ∈ E } . For node classification task , the goal of GNNs is to learn optimal mapping function f ( · ) to predict the class label yv for node v. Generally speaking , GNNs follows a framework including aggregation and combination in each layer . Different GNNs have proposed different ways of aggregation and combination . In general , the k-th layer of a GNN reads a ( k ) v = Aggregate ( k ) ( { h ( k−1 ) u : u ∈ N ( v ) } ) , h ( k ) v = Combine ( k ) ( h ( k−1 ) v , a ( k ) v ) , ( 1 ) where h ( k ) v is the output for k-th layer of node v , h ( 0 ) v is the input vector of node v . 2.2 LABEL PROPAGATION . In Label Propagation ( LP ) , node labels are propagated and aggregated along the edges in the graph ( Zhou et al. , 2004 ; Zhu et al. , 2005 ; Wang & Zhang , 2007 ; Karasuyama & Mamitsuka , 2013 ) . There are some works which were designed to improve the performance of label propagation . For example , Gong et al . ( 2016 ) proposed a novel iterative label propagation algorithm which explicitly optimizes the propagation quality by manipulating the propagation sequence to move from simple to difficult examples ; Zhang et al . ( 2020 ) introduces a triple matrix recovery mechanism to remove noise from the estimated soft labels during propagation . Label propagation has been applied in semi-supervised image classification task . For example , Gong et al . ( 2017 ) used a weighted Knearest neighborhood graph to bridge the datapoints so that the label information can be propagated from the scarce labeled examples to unlabeled examples along the graph edges . Park et al . ( 2020 ) proposed a novel framwork to propagate the label information of the sampled data ( reliable ) to adjacent data along a similarity based graph . Compared to these methods , we utilize the intrinsic graph structure instead of handcrafted graph to propagate clean labels information , which is more reliable for graph-structured data . Besides , GNNs are utilized by us to extract features and classify nodes for graph-structured data . 2.3 META-LEARNING BASED METHODS AGAINST NOISY LABELS . Meta-learning aims to learn not only neural networks ’ weights , but also itself , such as hand-designed parameters , optimizer and so on ( Andrychowicz et al. , 2016 ; Finn et al. , 2017 ) . Several works have utilized meta-learning paradigm to deal with label noise . For example , Li et al . ( 2019 ) has proposed to find noise-tolerant model parameters by keeping the consistency between the output of teacher and student networks , and Li et al . ( 2017b ) trains the teacher networks with samples with clean labels and then transfer the knowledge to student networks so that the student can learn correctly even if the existence of mislabeled data . Besides , Ren et al . ( 2018 ) ; Jenni & Favaro ( 2018 ) ; Shu et al . ( 2019 ) utilize meta-learning paradigm to re-weight samples , i.e. , weight samples with clean labels more and weight mislabeled samples less . The weighting factors are optimized by gradient decent or generated by a network to minimizes the loss on a small amount of samples with correct labels . In contrast , meta-learning paradigm is utilized in this paper to learn how to aggregate origin labels and pseudo labels properly . We can get more credible supervision by combining the original label information with the label information provided by LP properly . 3 METHODS . 3.1 PRELIMINARIES . Given a graph data with n nodes and their labels D = { ( x0 , y0 ) , ( x1 , y1 ) , ... , ( xn−1 , yn−1 ) } , where xj is the j-th node and yj ∈ { 0 , 1 } c is the label over c classes . Dtrain = { ( x0 , y0 ) , ( x1 , y1 ) , ... , ( xs−1 , ys−1 ) } are training nodes with noisy labels . Our goal is to enable the GNNs f ( xj ; w ) trained with noisy sets Dtrain can also generalize well on test nodes . w is the learnable parameters of GNNs . In our method , m nodes with true labels Dclean = { ( xs , ys ) , ( xs+1 , ys+1 ) , ... , ( xs+m−1 , ys+m−1 ) } in the graph are provided as the initial clean sets ( m s ) . GCN ( Kipf & Welling , 2016 ) and GAT ( Velickovic et al. , 2017 ) are utilized in our experiments to extract features and classify nodes . Our method includes two main parts : label propagation and label aggregation . We will go into details about these two parts in the following section 3.2 and section 3.3 . 3.2 LABEL PROPAGATION . Label Propagation is based on the label smoothness that two connected nodes tend to have the same label . Therefore , the weighted average of neighbor nodes ’ label of a node is similar to this node ’ s true label . An illustration of LP part in our method can be found in Figure . 1 . The first step of LP is to construct an appropriate neighborhood graph . A common choice is k-nearest graph ( Iscen et al. , 2019 ; Liu et al. , 2018 ) but there is an intrinsic graph structure ( adjacency matrix A ) in graph data , so our similarities matrix W with zero diagonal can be constructed with A , whose elements Wi , j are pairwise similarities between node i and node j : Wi , j = Ai , j d ( hi , hj ) + ε , ( 2 ) where hi , hj are the feature vectors extracted by GNNs for node i and node j. d ( · , · ) is a distance measure ( e.g. , Euclidean distance ) . ε is an infinitesimal . Note that we can get W with time complexity O ( | E | ) instead of O ( n2 ) because A is a sparse matrix whose edge lists are given . Then we can normalize the similarities matrix W : S = D−1/2WD−1/2 , ( 3 ) where D is a diagonal matrix with ( i , i ) -value to be the sum of the i-th row of W . Let Y ( k ) = [ y ( k ) 1 , ... , y ( k ) n ] T ∈ Rn×c be the soft label matrix in LP iteration k and the i-th row y ( k ) i is the predicted label distribution for node i . When k = 0 , the initial label matrix Y ( 0 ) = [ y ( 0 ) 1 , ... , y ( 0 ) n ] T consists of one-hot label vectors for i = s , s+ 1 , ... , s+m− 1 ( i.e. , initial clean sets ) or zero vectors otherwise . The LP ( Zhu et al. , 2005 ) in iteration k can be formulated as : Y ( k+1 ) = SY ( k ) , ( 4 ) y ( k+1 ) i = y ( 0 ) i , ∀i ∈ [ s , s+m− 1 ] ( 5 ) In Eq . ( 4 ) , every node ’ s label in the ( k + 1 ) -th iteration equals the weighted average of its neighbor nodes ’ labels in k-th iteration . In this way , the clean sets propagate labels to the noisy training nodes according to normalized edge weights . And then in Eq . ( 5 ) , the labels of clean sets nodes are reset to their initial values . The reason is that we can take full advantage of the tiny minority of clean nodes and in case that the effect of clean sets fade away . Co-teaching ( Han et al. , 2018 ) and Co-teaching plus ( Yu et al. , 2019 ) have been proposed to train DNNs robustly against label noise . There are two DNNs which select samples with small loss from noisy training sets to train each other . Our method is similar to theirs to some extent because LP is utilized by us to select true-labeled samples from Dtrain for training . However , instead of taking the nodes with small loss as true-labeled nodes , we select the nodes Dselect whose original labels are same with pseudo labels for training . Original labels ofDselect are credible and we also inject them to initial clean sets Dclean for better LP in next epoch . This is why our method can achieve better performance even if few true-labeled nodes are provided . 3.3 META-LEARNING BASED LABEL AGGREGATION 1 . Fo w ar d no is y 2 . Aggregation net forward 3 . Ba ck w ar d no is y Training loss 6.Backw ard on backw ard 4 . Fo rw ar d cl ea n 5.Backw ard clean Clean loss Gradients descent 7.Aggregation net backward In section 3.2 , the selected training nodes ( node 5,7 in Figure.1 ) have been utilized for training and LP but the left training nodes Dleft ( node 6,8,9,10 in Figure.1 ) with abundant information haven ’ t been fully exploited . In this section , we mine the abundant and precious information from Dleft via meta learning . The computation process of label aggregation is shown in Figure . 2 . For ∀ ( xj , yj ) ∈ Dleft , we can get two loss values : l1 = loss ( ŷj , yj ) , ( 6 ) l2 = loss ( ŷj , ỹj ) , ( 7 ) where ŷj is the label predicted by GNNs for training node j and ỹj is the pseudo label predicted by LP for node j . We can also get final label yj for node j by aggregating original label yj and pseudo label ỹj : yj = λjyj + ( 1− λj ) ỹj , λj ∈ [ 0 , 1 ] ( 8 ) where λ is the aggregation coefficient . Some previous methods designed a weighting function mapping training loss to sample weights for noisy label problems ( Kumar et al. , 2010 ; Ren et al. , 2018 ; Shu et al. , 2019 ) . Instead , we utilize a 3-layer multi-layer perceptron ( MLP ) as the aggregation network g ( · ; · ) to map loss values to aggregation coefficient λj : λj = g ( l1 ‖ l2 ; θ ) = λj ( θ ; w ) , ( 9 ) Where l1 ‖ l2 is a 2-dimensional vector which is the concatenation of l1 and l2 and θ is the weights of aggregation network g. The rationality lies on a consensus that samples ’ loss values are affiliated with the credibility of samples ’ original labels ( Kumar et al. , 2010 ; Shu et al. , 2019 ; Yu et al. , 2019 ) . The MLP or aggregation networks ’ input layer are 2 neurons and its output layer is one neuron , which can be an approximator to almost any continuous functions . The activation function of the last layer is sigmoid function to ensure that output λj ∈ [ 0 , 1 ] . We can get the training loss Ltrj for node j : Ltrj ( w , θ ) = loss ( ŷj ( w ) , yj ( θ ) ) , ( 10 ) Then we can backward on the GNNs : ŵt ( θt ) = wt − α | Dleft | ∑ ( xj , yj ) ∈Dleft ∇wLtrj ( w , θt ) |wt , ( 11 ) where α is the learning rate of GNNs . Then we can get the loss Lc on clean sets Dclean : Lc ( ŵt ( θt ) ) = 1 | Dclean | ∑ ( xi , yi ) ∈Dclean loss ( f ( xi ; ŵt ( θt ) ) , yi ) , ( 12 ) Where f ( xi ; ŵt ( θt ) ) is the output of GNNs . Then we can utilize Lc to update the weights of aggregation network : θt+1 = θt − β∇θLc ( ŵ ( θ ) ) |θt , ( 13 ) where β is the learning rate of aggregation network . Finally , GNNs ’ weights can be updated : wt+1 = wt − α | Dleft | ∑ ( xj , yj ) ∈Dleft ∇wLtrj ( w , θt+1 ) |wt . ( 14 ) To some extent , this part is similar to re-weight based methods ( Ren et al. , 2018 ; Shu et al. , 2019 ) . However , LPM has two significant advantages . Firstly , re-weight based methods can not remove the damages caused by incorrect labels because they assign every noisy training sample a positive weight while LPM potentially has the ability to take full advantage of noisy samples positively . Secondly , LPM can generate comparatively credible labels for other usages while re-weight or some other methods can not . Algorithm . 1 shows all the steps of our algorithm .

Summary: The paper proposes a robust training algorithm for graph neural networks against label noise. The authors assume the labeled nodes are divided into two parts, clean part without noise and train part with some noise. The proposed method contains two parts. Firstly, it leverages label propagation (LP) trained on the clean nodes to assign pseudo labels on train nodes with noisy labels. Secondly, the authors design a learnable weight \lambda to learn the label for those noisy nodes where LP does not agree with the original labels. The final graph neural network is trained with clean nodes, high confidence train nodes, and uncertain train nodes with learned labels. The authors conduct experiments on four graph datasets with manual injected noise and one real-world noisy dataset to validate the proposed method.